The 63-year-old comedian was asked to fill in for the TV legend – who hosted the game show in two different spells – when the late entertainer once fell ill.
But after Jim had appeared on the series he claimed show bosses later used him as a “chess piece” to sign him up as a cheaper alternative to Bruce, who passed away in August aged 89.
Speaking on an upcoming episode of Piers Morgan’s Life Stories, Jim said: “One day there was a knock on the door and a man said, ‘Jim, we are recording The Generation Game tomorrow and Bruce has gone sick. Can you step in to do it?’
“I said I would give it a go. I was not particularly good, but it made Bruce get better really quickly. When negotiations came for Series 34 for Bruce they used me as a chess piece to say, ‘We can’t go to that [his fee], Bruce. We have this guy.’
“I took over and I don’t think Bruce was very happy. He did not speak to me again for the rest of his life.
“He did not speak to me much anyway. I was always the upstart.”
Bruce hosted The Generation Game from its first episode in 1971 to 1977 before Larry Grayson took over.
He later returned to front the programme in 1990 and remained until 1994 when Jim replaced him in 1995, and he hosted the show until 2001.
Speaking to Piers, he added: “So I did it and I loved it. It was fantastic.”
Following Jim’s stint as the presenter, Graham Norton and Vernon Kay have also since hosted The Generation Game for short spells, and it is set for a reboot with comedy duo Mel Giedroyc and Sue Perkins.
Speaking about the revival, Jim added: “It was of its day. I would rather watch Ant and Dec do it.”
Generation Game bosses are set to include a tribute to Bruce in the rebooted series.
A source recently said: “The plan is to have Bruce suitably honoured in the first episode – almost as if he’s handing over the reins. He’s such a huge part of the show that it seems the right thing to do. The plan is for him to feature on screen.
“The final details haven’t been ironed out but it’s going to be a touching moment, a big tribute to the man who started it all.”
Read more from Yahoo Celebrity: