And according to a new survey, those opinions are very negative - even for a politician.
Research conducted by Savanta ComRes exclusively for Yahoo News UK this week found the most common word people use to describe Hancock is: "Idiot".
There were also multiple mentions of words too offensive to print out in full - but can be seen in the word cloud below.
The former health secretary, who is still an MP, entered the jungle on day two of this year's show.
His decision, weeks after he announced he is writing a book about his time as health secretary called Pandemic Diaries - triggered backlash from politicians and the public, with the Tory party swiftly removing the party whip.
Rishi Sunak told The Sun he was "very disappointed" in the former health secretary's decision.
"Politics is a noble profession, at its best - it can and should be - but it's incumbent on politicians to earn people's respect and trust," the PM said.
"They do that by working hard for their constituents, as the vast majority of MPs do - that is why I was very disappointed with Matt's decision."
Labour leader Sir Keir Starmer said Hancock "should voluntarily leave and get back to parliament and do his job".
Savanta asked 2,132 people between 18-20 November: "How would you describe Matt Hancock in one word?".
Alongside "idiot", other words among the top 10 included "bad", "tw*t, "stupid", "useless", "silly", and "weird".
One respondent said: "[I] don't think I'm allowed to say what I think of him, it's not very polite."
Another said: "Should be back in Westminster looking after his constituents."
'A bit of forgiveness'
Shortly after entering the jungle, Hancock said he was looking for "a bit of forgiveness" on the matter after being challenged by a fellow camp mate on the scandal.
Many, however, were not sympathetic with one campaign group flying a banner reading "COVID bereaved say get out of here" over the I'm A Celeb camp.
Another group, COVID-19 Bereaved Families For Justice, have also condemned Hancock's appearance on the show. "He isn't a 'celebrity', he's the former health secretary who oversaw the UK having one of the highest death tolls in the world from COVID-19 whilst breaking his own lockdown rules," it said.
Despite this apparent wave of negativity, Hancock has survived four eliminations on I'm A Celeb, with the public also voting him as leader of the camp.
This is reflected in the polling, with around a third of the top 10 words in the poll being more sympathetic - with "good", "funny", and "okay" making the cut.
One respondent described him as a "well-educated MP who made a silly mistake but on the whole was good during the pandemic".
Hancock was exposed as having broken the COVID social distancing guidance he himself had set while having an affair with aide Gina Coladangelo during the pandemic.
He initially tried to resist calls for his resignation, but stepped down in June last year.
Hancock's COVID policy on care homes was ruled "unlawful" by the High Court in April after the government discharged older patients into care homes without testing at the start of the pandemic.
He has also been forced to deny accusations that he had helped his local pub landlord, Alex Bourne, obtain a multimillion-pound contract to supply test tubes for NHS COVID testing.
And on Wednesday, parliament's anti-corruption watchdog said Hancock had breached rules after he failed to announce he was appearing on I'm A Celeb and Channel 4's SAS Who Dares Wins last month.
The top 20 most common words people polled used to described Matt Hancock:
Watch: Matt Hancock broke government rules over I'm A Celebrity... Get Me Out Of Here! stint, watchdog says