The actor left his role as Mick Carter in the BBC One soap at Christmas after nine years, bowing out in dramatic fashion as he disappeared into the sea.
But while Dyer said he would be eternally grateful to EastEnders and would miss plenty of people, he added that there were a few he didn't like at all.
Appearing on ITV1's The Jonathan Ross Show, Dyer was asked by Ross: "Who will you miss the least?"
Dyer replied: "There’s a few of them actually, I ain’t going to lie. I won’t say [who], you know. Most of them I love dearly. Not all…"
Ross asked: "So they know who they are, though?"
Dyer said: "I would have thought so."
Read more: Pointless announces new guest hosts
He also added that his final scenes had not just been traumatic because of the emotional exit - but also because of the broccoli water he was plunged into to act as the sea.
On leaving, he said: "I tell you what it is, I’m lazy. It’s f***ing hard. Thirty pages you’ve got to learn a day. It’s like a film a week.
"My final scenes were in the sea, in Ramsgate at 2am. We did the exteriors in the sea and the interiors you go to a water tank. But EastEnders couldn’t get Pinewood, so we had to go to Basildon.
"It’s all warm and lovely but to make it look like the sea they put broccoli juice in it. You smell of a***. It’s f***ing broccoli juice. Warm broccoli juice. It was very emotional, but I stunk."
However, Dyer added that there were plenty of people he missed on the soap and was pleased that the door had been left open for a potential return.
He said: "If it goes t**s up I can go back. I can turn up in three years’ time, walk in the Queen Vic smothered in seaweed…
"They wanted to keep it open. I’m very privileged and grateful for that job. It really did change my career and my life."
Dyer appears on this week's Jonathan Ross Show, which airs on Saturday at 9.40pm on ITV1.