Mr Hammond told cabinet ministers Britain would be unlikely to win any legal battle regarding the bill to settle its outstanding liabilities, the Daily Telegraph reported.
According to the paper the chancellor said the country would only save between £3bn and £9bn if negotiations end in no-deal.
“He said that the Treasury’s legal advice was that if we left without a deal we would still have to pay the EU £30-36bn because we would be unlikely to win any case that went to international arbitration,” a cabinet source reportedly said.
It comes as Theresa May travels to Brussels on Wednesday for what has been called the “moment of truth” for Brexit negotiations as she battles to keep hopes of securing a deal alive.
The prime minister is set to address EU leaders amid growing concerns the two sides will be unable to bridge the gap over the issue of the Irish border.
Her meeting comes after hastily arranged talks broke down at the weekend meaning negotiations are once again deadlocked.
It had been hoped that the meeting would see leaders of the remaining 27 member states give the green light for a special summit in November to finalise the terms of Britain's withdrawal.
However European Council president Donald Tusk has warned that without new "concrete proposals" from the British to break the logjam over the so-called Irish border "backstop", further progress may be impossible.
With her party split - and some Tory MPs openly calling for her to go - Mrs May has little room to manoeuvre if she is to secure a deal which stands any chance of getting through Parliament.
Ahead of her visit to Brussels, Mrs May was able to secure the backing of her cabinet, for now, amid reports that some Brexiteer ministers were prepared to quit if she gave too much ground to Brussels.
During a marathon three-hour meeting on Tuesday, she insisted she would not accept an agreement on the backstop - intended to maintain an open border between Northern Ireland and the Republic - which undermined the integrity of the UK or tied it indefinitely to EU customs arrangements.
The Prime Minister will briefly address the leaders of the EU 27 on Wednesday evening before they discuss the state of play in the Brexit negotiations over a working dinner while she leaves.
Her official spokesman said she would take the opportunity to set out the areas where progress had been achieved while stressing her commitment to finding an agreement.
"We want to secure a deal as quickly as possible. We think it is in the best interests of the UK and European Union to forge that deep future partnership," the spokesman said.
With hopes of a November summit fading, focus has turned to the next scheduled meeting of the European Council in December as the last chance to secure a deal and get it ratified by the UK and European parliaments before Britain leaves in March 2019.
Additional reporting by Press Association