The 47-year-old host of the ITV talent show – who has presented The X Factor for 12 series – admitted that in 2015 he had gone to Cowell’s house to confront him about not being happy with his role on the show.
O’Leary told The Sun: “I’ve had my state-of-the-union meeting with Cowell as well. You’re watching the show one day and going, ‘I’m not in any of this’.
“He does that great Simon thing where you walk into his house and you’re like, ‘Stick to the plan, say what you want to say,’ and then you sit down, you’re going through it and he goes, ‘Do you want some chicken pie?’
“And in your head, you’re like, ‘Actually, I really like chicken pie’.
“The most delicious chicken pie you’ve had in your life turns up and you have this thing and you go, ‘I don’t think I’m in the show enough.’ He goes, ‘Yeah, I totally agree. Yeah, absolutely.’ Then you go, ‘Great, now I can just have a great time,’ because he’s great company.”
The BBC Radio 2 DJ’s comments come after Ant McPartlin and Declan Donnelly revealed in their new autobiography that they flew to Los Angeles in 2012 to tell Cowell they were quitting Britain’s Got Talent as they felt they had been cut out of the show.
In their new book Once Upon A Tyne, the Geordie duo revealed: "The audition days had been long, which in itself we don't mind, but after filming interviews and interacting with every single act, as well as making jokes, giving reactions and everything else we always do, we found that when the show went out on TV, we hardly seemed to feature in it at all.
"We began to feel we were wasting our time doing the auditions because all the work we were doing was ending up on the cutting room floor.
"At that stage, it felt like it may as well have been anyone hosting the show and we said when our current contract expired, we thought it was time to move on.”
But Cowell managed to persuade them to change their minds.
Donnelly said: "He made a promise to us: that things would change and we'd never feel like that again. He desperately wanted us to stay.”
And the pair are still hosting Britain’s Got Talent after 14 series.
O’Leary took over hosting The X Factor from Kate Thornton after three series in 2007.
In 2015 he was replaced by Olly Murs and Caroline Flack for one series. But O’Leary returned to host the singing talent show in 2016, and is still currently the presenter.