The then-Tory backbencher, who backed the Leave campaign, was heard making the remark outside the Groucho Club in Soho, according to the Evening Standard.
Joy Lo Dico, the journalist behind The Londoner diary column at the time, said she found Mr Kwarteng talking “feverishly” into his phone after the result. “The markets are going mad,” he told her.
The Brexiteer was later heard saying: “Who cares if Sterling crashes? It will come back up again.”
Mr Kwarteng is contending with massive market turmoil sparked by his tax-cutting mini-budget, as a disastrous day for the pound saw Labour take its biggest poll lead over Tories for more than 20 years.
The pound steadied in early trading on Tuesday as it recovered slightly from the record low of 1.0327 against the US dollar on Monday after traders were panicked by Mr Kwarteng and Liz Truss’s economic plans.
Lenders were withdrawing some of their mortgages on Monday as uncertainty reigned in the wake of Mr Kwarteng’s £45bn package of tax cuts set out on Friday.
Senior Tory MP Huw Merriman – who backed former chancellor Rishi Sunak for Conservative leader – warned that Ms Truss may be losing voters “with policies we warned against”.
One senior Tory backbencher told The Independent that combination of the “disastrous” mini-Budget and plummeting pound had “substantially reduced” the Tory party’s chances of winning the next election.
“It’s been deeply damaging to our reputation on the economy – but that’s what happens when you have ideologues running the show. They’re like tea party Republicans or right-wing Corbynistas,” said the MP.
The senior figure said fellow moderates felt there was no point trying to change leader because it would make the Tory party an “absolute joke” to the wider electorate, and see another right-winger Tory grassroots. “What’s the f****** point?” they said.
The Tory added: “The mood now is that the next general election cannot be won, so some of us are thinking about how you reshape the party after a defeat.”
Another Tory MP said the mini-budget had badly damaged the party’s next election. “It’s done our chances no good at all. The bankers bonus thing and scrapping 45p rate were very strange and stupid. They’re just going to wind people up in the red wall.”
Gerard Lyons, the radical right-wing economist who has advised Ms Truss, admitted that axing the top rate tax had “spooked” the markets – saying he was confused by the move.
“I was against those unexpected and unexplained small tax changes on Friday,” Mr Lyons told Channel 4 News. “Even though in the scale of things they were small compared to the reversal of the tax increases, they spooked the markets.”
He added: “I think what we need to do, given both the Treasury and the Bank of England statements, is see how the market settles down in coming days.”
The Treasury said the chancellor would announce a “medium-term fiscal plan” to start bringing down debt levels on 23 November.
And the Office for Budget Responsibility (OBR) will publish its updated forecasts for the current calendar amid widespread criticism that there was no update when Mr Kwarteng set out his “growth plan” last week.
At one point, it was thought that the Bank of England would be forced to step in with an emergency interest rate hike amid fears the pound could drop to parity with the dollar.
However, governor Andrew Bailey said the monetary policy committee, which sets interest rates, would make a full assessment of the impact on inflation and the fall in sterling at its next scheduled meeting in November and then “act accordingly”.
Shadow health secretary Wes Streeting said on Tuesday that the “cavalry is coming” as the latest YouGov poll shows Labour ahead 45 per cent top 28 per cent.
He added: “We’re got serious people, with a serious plan that would make an enormous difference to families right across the country and to businesses, who are the backbone of our economy and will be the bedrock of economic growth.”