Philip Hammond’s budget cannot erase the stain on the soul of his party | Rafael Behr

Philip Hammond’s budget faces political death by impossible demands. The question is how long he can defer the execution. The chancellor is supposed to maintain fiscal discipline and ease the pain of austerity. He must build houses, but not in anyone’s backyard. He must raise revenue, but not from taxes.

And those are just pressures from inside the cabinet. If Hammond makes it through to the weekend without colleagues calling for him to be sacked, he’ll have beaten the odds.

The chancellor’s comfort is that hardly anyone beyond Westminster is paying attention. Opinion polls are not tracking headline volatility. Labour is consistently ahead by a few points, but the Tories have not been dramatically penalised for weeks of scandal and Brexit turmoil. Vote shares have hardly shifted since the election.

A plausible explanation for this apparent stalemate is provided by Andrew Cooper, a Tory peer and co-founder of the Populus polling agency. He describes voting intention numbers as “a meaningless answer to a stupid question”. Cooper’s research finds voters thoroughly disengaged, “even more so than usual”. They see few attractive options and don’t want to be made to choose. Fatigued by a surfeit of argument, the audience has tuned out. Asked what they’d do in an election, most people default to whatever they did last time.

This is not encouraging for the Tories, for whom last time was traumatic. They gained votes but lost seats. That was the result of a strategic blunder exacerbated by a useless campaign. Theresa May made a bet that people who voted to leave the EU, many in Labour constituencies, would automatically endorse the more Brexity of the two main parties. Many did. But it didn’t occur to the prime minister that people who voted remain might be exercised by the same impulse in the opposite direction.

A hard Brexit would take Britain out of the EU’s single market and customs union and ends its obligations to respect the four freedoms, make big EU budget payments and accept the jurisdiction of the ECJ: what Brexiters mean by “taking back control” of Britain’s borders, laws and money. It would mean a return of trade tariffs, depending on what (if any) FTA was agreed. See our full Brexit phrasebook.

May also underestimated the residual potency of raw Tory hatred in parts of the country. That tribal habit, combined with calculated ambiguity in Labour’s European position, meant too few “deep red” leavers rallied to May’s hard Brexit banner, while “light blues” – the type who once backed Tony Blair but switched to David Cameron in 2010 and 2015 – were repelled.

On that basis Jeremy Corbyn’s commitment to Brexit, expressed in votes for the Tories’ withdrawal bill, ought be driving support away from Labour. Perhaps, below the radar, it is. But the fate of amendments in parliament is also the kind of detail no one wants to know.

The nation’s eyes glaze over when conversation turns to EU technical arrangements, the single market, tariffs, treaties. That was true before and during the referendum campaign. It is true today. Britain was briefly excited by an existential question about its future, but the subsequent process interests only a tiny minority.

A soft Brexit, while not officially defined, would keep Britain in either the single market or the customs union or both. It could be achieved along the lines of the Norway model (see EEA/EFTA) or via an FTA, but would require concessions on free movement, ECJ jurisdiction and budget payments. Brexiters do not consider a soft Brexit as really leaving the EU. See our full Brexit phrasebook.

This helps explain why Brexit polling also lacks volatility. Pro-Europeans brandish reports of economic gloom in the hope that scales will fall from leavers’ eyes. Leavers hear only unpatriotic revelling in pessimism, and disdain for the will of the majority. This week’s report of the relocation of the European Banking Authority to Paris and the European Medicines Agency to Amsterdam is mourned by one side as a forfeit of national status; the other side dismisses it as a necessary evacuation of Brussels’ bureaucrats. And that is just those who even notice.

That doesn’t mean remain and leave camps are thinly populated. The divide is significant, but more as rival sets of values than as preferences for the terms of UK-EU trade. One large-scale poll, conducted after the referendum, tested social attitudes on a range of broad political propositions – whether the world has been made better by immigration, globalisation, feminism, environmentalism, the internet.

Those who saw such things as forces for good were much likelier to have voted remain. Leavers showed the reverse tendency. A repeat of the exercise after this year’s election found the same fault line predicting party allegiance. Anyone under 45 was likely to be in the liberal-remain trench in this culture war – so not a Tory.

Brexiters are not all modernity-hating pensioners, of course. There are young and leftwing Eurosceptics. There is also a breed of Tory that sees no contradiction between social liberalism and flight from the EU.

But that is a relatively niche combination. It just happens to have disproportionate representation at the top of the Conservative party, in figures such as Boris Johnson and Michael Gove. They might act as if the referendum was won on their specific terms, but anyone who witnessed the leave campaign knows it mobilised nostalgic illiberalism. Johnson and Gove didn’t share platforms with Nigel Farage, but they entered a Faustian pact to borrow his supporters. That alliance has left its mark on the Tory brand.

The post-referendum collapse in Ukip’s vote share is unlikely to describe a sudden evaporation of hostility to immigration and hatred of gay marriage. Ukip lost its defining Eurosceptic cause, but as a menu of values, Faragism survived. It was swallowed whole by the Tories, bloating their vote share in some places, and causing political indigestion in others.

Nothing the chancellor does in his budget can help with that ailment. There are no tax breaks or housebuilding programmes to change the minds of people who think Conservatives stand in cultural hostility to the kind of world they want to live in. The problem is not policy but a stain on the soul of the party.

No one can safely predict how this plays out in an election. Some Tories believe Labour’s position has been flattered by protest votes that will melt away if it looks like Corbyn might actually become prime minister. Some hope that referendum resentment will fade once Brexit is a done deal.

But that misreads what it now means to be a remainer. The label comes from one ballot, but the voters it describes were not invented by that campaign, nor did they vanish after it. Their concerns were never defined by EU treaties. Yet May took the referendum result a licence to ignore them, and governs as if she despises them. Those choices will not be quickly forgotten.

• Rafael Behr is a Guardian columnist