The broadcaster, 66, began hosting the sports quiz show in 1997 and left in 2021.
Appearing on BBC Breakfast, she suggested the corporation had wanted her to say she was stepping back by her own choice, and not that she was being replaced as part of a “refresh”.
“It is such a shame because, I have to say, that the BBC had told us we were going,” she said.
“They wanted to refresh the programme and that is absolutely fine. Everyone has the right to do that. We don’t own the programme.”
The former tennis pro went on: “I had had 24 amazing years working with the most incredible people. So we knew it was going to happen and it was just the way in which it happened and the way it was handled, and the way the BBC sort of wanted me to say that I was walking away from it.
“And yet, I would never walk away from a job I love. I don’t mind being replaced. Absolutely fine. That happens. But it was just the way it was handled.
“I think we regret the way it was handled. I think if we look back on it we could have handled it better. I think the BBC could have handled it better."
Barker said it was "such a shame" as she "loved" the show.
"And I don’t look back on it badly just because of a couple of bad days and then some negative publicity, which I think affected quite a lot of people," she added.
It was announced in September 2020 that Barker was stepping down from Question Of Sport.
It was later confirmed Paddy McGuinness would be taking over the role, alongside the show’s team captains Matt Dawson and Phil Tufnell, who were replaced by Sam Quek and Ugo Monye.
Barker has since announced that she is also stepping down from presenting coverage at the Wimbledon tennis championships.
She later suggested that what happened with Question Of Sport led to her choosing to retire from Wimbledon before she could be asked to leave.
Read more: Sue Barker on Cliff Richard fling
Speaking on Desert Island Discs this year, Barker said: “The way it was handled made me think about Wimbledon and the way I wanted to walk away on my own terms rather than be pushed out the door.”
Additional reporting from PA.
Watch: Emotional Sue Barker receives standing ovation during Centre Court celebration