In a letter to Boris Johnson, he said the Government “owe it to people who have sacrificed so much in this pandemic to be honest when we have let them down as I have done by breaching the guidance.”
He added: “The last thing I would want is for my private life to distract attention from the single-minded focus that is leading up out of this crisis.”
In response, the Prime Minister wrote: “You should leave office very proud of what you have achieved – not just in tackling the pandemic, but even before Covid-19 struck us.”
Mr Johnson had refused to sack Mr Hancock, with his spokesman saying the PM considered the matter closed after receiving the West Suffolk MP’s apology on Friday.
Lawyers described how Mr Hancock may have broken the law regarding coronavirus restrictions, although he admitted only to breaching guidance.
There were also questions about Mrs Coladangelo’s appointment to her role in the Department of Health and Social Care (DHSC) in the first place.
The Prime Minister resisted calls to sack Mr Hancock, who said he was “very sorry” for letting people down after The Sun first reported he was having an extramarital affair.
But Tory Duncan Baker said it was not enough before Mr Hancock quit.
Mr Baker told his local newspaper, the Eastern Daily Press: “In my view people in high public office and great positions of responsibility should act with the appropriate morals and ethics that come with that role.
“Matt Hancock, on a number of measures, has fallen short of that. As an MP who is a devoted family man, married for 12 years with a wonderful wife and children, standards and integrity matter to me.
“I will not in any shape condone this behaviour and I have in the strongest possible terms told the Government what I think.”
Veteran Tory MP Sir Christopher Chope joined the growing list of Conservatives calling on Mr Hancock to resign, saying his constituents were “seething”.
The MP for Christchurch told BBC Radio 4’s PM programme that the annual general meeting of his local party association had “unanimously called on Matt Hancock to resign immediately” and he said he felt this reflected the mood of the public.
He said: “They felt that he was in breach of the ministerial code; that he was in breach of the lockdown regulations; that he is the person who has been passing the laws, signing off the regulations, requiring people to comply with restrictions upon their freedom – many of which people don’t agree with – but they’ve complied with out of respect for the rule of law.
“And they’re absolutely seething now, that having made sacrifices themselves, including not seeing loved ones and all the rest of it, they find that the person who was in charge of all this feels free to not comply with his own regulations and I think that’s the essence of it really.”
He added: “I agree with that. I think that his position is untenable. For that reason, the sooner he does the honourable thing and announces his resignation, the better – because otherwise it is not going to go away.
“This is going to run and run, and it will impact adversely upon all of those of us who are involved in public life who are trying to set an example.”
Former Cabinet minister Esther McVey, the MP for Tatton, told GB News: “If it would have been me, I would have resigned myself, and I said that for Dominic (Cummings), and I’m hoping that Matt Hancock is thinking the same thing, that he doesn’t have to have it pushed upon him.”
She said it would be “viewed far more admirably” if he resigned.
Conservative MP William Wragg tweeted: “Re Mr Hancock, a thought: Covid regulations have created a dystopian world of denunciation, finger-wagging & hypocrisy.
“Let us be freed from this tyranny of diktat and arbitrary rule. As we shall inevitably see with this sad example, the revolution always consumes its own.”
It comes as a snap poll by Savanta ComRes, released hours after photographs of the pair kissing in Mr Hancock’s ministerial office surfaced, found 58 per cent of UK adults feel he should resign, compared to 25 per cent who say he should not.
The Covid-19 Bereaved Families For Justice group, which represents those who have lost loved ones to the pandemic, also called for Mr Hancock to go.
In a letter to the Prime Minister, the group said it has broken its “position of neutrality on ministerial conduct” to urge Mr Johnson to relieve Mr Hancock of his job.
The SNP said Mr Johnson “risks jeopardising vital public health measures” by retaining Mr Hancock as Health Secretary.
The party said there are “very serious questions” for Mr Hancock and the incident cannot “simply be brushed under the carpet”.
Its Westminster deputy leader Kirsten Oswald said: “The Prime Minister must at long last do the right thing and put his responsibilities to public health first.
“There must be public confidence in those setting the rules and it cannot be the case that it is one rule for the Tory elite and another for the rest of us.
“Despite declaring the matter closed, the reality is that it is anything but closed.
“There are very serious questions for Matt Hancock to answer over the appointment of his aide to the lucrative position, as well as questions over whether or not Hancock broke the ministerial code.”
In a statement on Thursday, Mr Hancock said: “I accept that I breached the social distancing guidance in these circumstances, I have let people down and am very sorry.
“I remain focused on working to get the country out of this pandemic, and would be grateful for privacy for my family on this personal matter.”
A Downing Street spokesman said Mr Johnson had accepted Mr Hancock’s apology and “considers the matter closed”.