Baker played the timelord in the BBC show from 1974 to 1981 and is regarded by many fans as the finest Doctor.
In an interview to mark the 60th anniversary of the series, he said that a celebratory get-together with the other Doctors was not on the cards.
“I avoid them, you know. Not with any malice. A degree of contempt, perhaps. But mildly. Mildly contemptuous,” he told Radio Times.
Told that three of the actors who played Tarzan - Ron Ely, Buster Crabbe and Johnny Weissmuller - regularly lunched together, Baker, 89, replied: “Unthinkable! Fancy being around a table with old Doctor Whos! Though I’d quite like to walk into a restaurant and find 12 of them around the table - I’d enjoy doing that double take, as I desperately tried to remember who they were.
“I could keep that going for hours. Oh, I can feel the malice rising.”
He added: “Inevitably, comparisons are made, and I find them odious.”
In 1983, Baker declined to pose with four of the other actors for a 20th-anniversary special named The Five Doctors. The producer had to use a Madame Tussauds waxwork of Baker in the promotional photographs.
He did return for an appearance in a 50th-anniversary special, The Day of the Doctor, in 2013.
The 14th Doctor, who will be seen for the first time this Christmas, is played by Ncuti Gatwa. Baker described him as “a handsome young man” with “marvellous cheekbones”.
The interviewer described Baker as having a “mischievous glint in his eye” as he said: “Nobody failed as Doctor Who, did they? Well, we mustn’t be ungenerous to anyone. But certainly I didn’t.”
Russell T Davies, who relaunched Doctor Who in 2005 and recently returned as showrunner, has ascribed his passion for the show to the Tom Baker era.
He once said: “I have a crucial memory of TV Comic’s weekly Doctor Who strip printing a gorgeous piece of artwork - drawn, I now know, by Gerry Haylock - showing Tom Baker in full hat, scarf and toothy grin.
“And something clicked in my head. Something clicked and has stayed clicked ever since. A simple thought which said: I love you.”