Cowell commented for the first time, admitting Walliams remarks were against the "ethos" of the ITV family entertainment show but said he had not been aware of them at the time.
The BGT creator and head judge told The Sun: "I didn’t really get involved in the decision. From what I understand, he decided to take a step back.
"It all happened relatively quickly. I was on holiday and then the next thing is I’m having conversations about who’s going to take his place."
"We did say to the two producers: ‘You have to make the final decision’ because, yeah, I am an exec producer, but they’re more hands on.
"Obviously everyone had some input - the network, the producers Fremantle, all of us said at the time, these comments are completely unacceptable.
"I had not heard them before and, yeah, it was upsetting. But from where I’m sitting, this wasn’t something I believe was constant."
Walliams apologised after The Guardian newspaper revealed comments he had made about BGT hopefuls during auditions recorded in January 2020.
The Little Britain star said in a statement: “I would like to apologise to the people I made disrespectful comments about during breaks in filming for Britain’s Got Talent in 2020.
"These were private conversations and — like most conversations with friends — were never intended to be shared. Nevertheless, I am sorry.”
The Guardian said it had seen a leaked transcript of a conversation in which Walliams referred to one contestant as a “c***”.
It reported that the contestant had bantered with the panel and when he walked off stage Walliams is said to have used the offensive word to describe him three times.
He reportedly said of a female trying out: “She’s like the slightly boring girl you meet in the pub that thinks you want to f*** them, but you don’t.”
“She thinks you want to f*** her, but you don’t."
The comments were apparently picked up by microphones.
Watch: Alesha Dixon says new BGT judge Bruno Tonioli doesn't understand the rules