Ministers are frustrated that the taxpayer is subsidising the costs of putting thousands of travellers up in hotels for 10 days when they "should not be travelling to red list countries".
About 30,000 people have already stayed in the hotels at a cost of up to £1,750 per person after arriving back from red list destinations. The price also includes PCR tests on day two and day eight.
Grant Shapps, the Transport Secretary, voiced his frustration in the Commons when he insisted the scheme did not make a profit for the Government but was subsidised.
"I also want to point out that people should not be travelling to red list countries. The only people who should be coming back to government quarantine are British or Irish citizens, or people with permanent rights of residence," Mr Shapps said.
"And there should be a limit to the number of people who are still abroad and wishing to return. I sometimes come across cases where people are still using the red list as if it is a case of 'It’s ok, I can come back and hotel quarantine'. That should not be the case."
It comes as the Government is abeing forced by threatened legal action to offer "waivers or fee reductions" to travellers from red list countries if they can show they would suffer financial hardship by having to stay in them.
Officials are expected to publish the details of the new fee cut scheme within the next month after legal action by the law firm PGMBM claimed the charges discriminated against those on low or no income, who could not afford to travel as a result.
Among those supported by the firm was a teenager whose GCSE studies were disrupted by hotel quarantine, which had stranded her in Portugal with her family unable to afford the cost.
Paul Charles, the chief executive of travel consultancy The PC Agency, said: "Increasing the charges is further tightening of the borders and making it more expensive for anyone thinking of coming from a red destination.
"The number testing positive in hotels is tiny. The principle of quarantine is a relic of earlier days. It would be better for the Government to replace it with testing on arrival."
It comes as Croatia became the latest country to impose obligatory Covid tests for all visitors from the UK, including those who are vaccinated, because of the surge in cases of the delta variant. Only children under 12 will be exempt, and the measure takes force on July 26.
In a statement, Croatia's state health institute said: "Due to the worsening of the epidemiological situation in the United Kingdom, Cyprus and Russia, as an additional measure for people arriving from those countries, a negative (Covid-19) test will be required regardless whether they are vaccinated or have already had a disease."