Matt Hancock is facing a backlash in his constituency over breaking social distancing rules, with a Conservative councillor calling for the former health secretary to be deselected if he does not stand down.
Mr Hancock, who has been MP for the constituency since 2010 and has a majority of more than 23,000, quit at the weekend after leaked CCTV footage showed him kissing his aide in his office in breach of COVID-19 rules.
The West Suffolk councillor said Mr Hancock's actions were "beyond the pale", adding that his "honour, integrity, probity and honesty, should he have had any, [is] trashed beyond redemption".
In his email to the chair of the West Suffolk Conservative Association, Mr Houlder said the controversy had shown Mr Hancock to be a "selfish, egotistical man".
"He has let every member of the public down, pontificating that they should all make huge sacrifices on the altar of the pandemic, whilst doing the complete opposite himself," Mr Houlder added.
Watch: Brexit - Why are performers upset with the rules on touring after the UK's departure from the EU?
He has also written directly to Mr Hancock, telling him: "If you have a shred of integrity or honour you would resign without delay.
"Do the residents of West Suffolk a favour and stand down."
Boris Johnson faced questions about the row in the Commons earlier on Wednesday.
The prime minister defended not immediately sacking Mr Hancock when the images emerged.
And he suggested the controversy was a "Westminster bubble" matter, something that was seized upon by Labour leader Sir Keir Starmer.
Mr Johnson used the phrase, often used to describe events within politics that have little impact or garner little interest beyond Westminster, in his response to Sir Keir asking him about the case of 27-year-old Ollie Bibby.
He died of leukaemia on 5 May, the day before the pictures of Mr Hancock were taken, with Sir Keir telling MPs how only one person was able to be at Ollie's bedside when he passed away.
The prime minister's spokesman later told journalists that Mr Johnson was not referring to the death of Mr Bibby when he made his "Westminster bubble" comment.