Banks could start making plans to move operations out of the UK as soon as Article 50 is triggered - if they have no assurance of an interim deal - London mayor Sadiq Khan has warned.
That would be a "catastrophe" for the sector, he said.
Theresa May is expected to trigger Article 50 later this month, beginning the two-year process of leaving the EU - and talks to establish the UK's future relationship with the bloc.
The Government has come under pressure to ensure that the UK and the EU come to a free trade agreement avoiding tariff and other trade barriers.
However, Mrs May has said she would rather walk away from talks than agree to a "bad deal".
City-based financial services firms are worried about the impact of losing "passporting" rights which currently enable them to sell their services freely across Europe.
Mr Khan called on Mrs May to ensure there was an early "interim deal" on transitional trade arrangements and warned banks could not wait for the two-year period to be concluded.
:: Brexit trade deal 'imperative' for UK growth plans, peers warn
He predicted a "huge" impact on jobs if passporting arrangements are lost and that without clarity on this companies could follow one another out of London.
Mr Khan said: "If, in the next few weeks - post Article 50 being served - they have not got the reassurance that there is going to be an interim deal in two years and one day, they will start making plans to move some of their operations.
"They don't want to leave.
"They love being in London, because of the technology, the talent, the finance, the legal services, our courts.
"They love that, but they will have no choice but to go, they have told me."
Mr Khan, who backed Remain the referendum campaign last year, acknowledged that an immediate flight of jobs from London after the Brexit vote had not materialised.