The actor, 72, is best known for playing the sitcom’s Marlene Boyce, who had a South London accent.
But off screen she sounds very different.
When Holderness appeared on the ITV show on Friday (10 September), panellist Jane Moore asked if her real accent surprised people.
And the star admitted: “It does upset people that I'm posh!”
The actor also entertained the panellists with the way she said the word "a***" in her well-spoken tones.
Recalling how people used to act around her, Holderness said: "When the show was going out I would often find in the middle of the supermarket that strange people would come and pinch my a*** because they had just seen it on the television.
"Usually I was quite polite and sometimes I'd say, 'Honestly, this is not appropriate'.
"I don't think any man would dare do it now, would they?"
Moore chimed in: "I don't know if we are allowed to say that word on the television but you say it so beautifully Sue, that maybe we'll get away with it."
As the other panellists laughed, Holderness offered to say it again.
The actor played Marlene Boyce in both Only Fools And Horses and its spin-off The Green Green Grass.
Her other television credits include Still Open All Hours and The Sandbaggers and she has also had many stage roles.