Mr Khan told a House of Commons committee that banks were telling him they wanted assurances that they did not face a cliff-edge move to World Trade Organisation tariffs.
This would be a "catastrophe" for the sector and could have a “huge” impact on jobs in the capital, he added.
Theresa May is planning to trigger EU withdrawal talks under Article 50 later this month.
Mr Khan called on her to strike an early "interim deal" on transitional trade arrangements with Brussels, warning that banks "can't wait" for the full two-year negotiation to be concluded.
Addressing the Commons Exiting the EU Committee, Mr Khan said without clarity on "passporting" arrangements, which allow UK-based financial institutions to operate across Europe, there was a danger of companies following one another in making the move away from London.
"If, in the next few weeks - post Article 50 being served - they have not got the reassurance that there is going to be an interim deal in two years and one day, they will start making plans to move some of their operations," Mr Khan told the cross-party committee.
"They don't want to leave. They love being in London, because of the technology, the talent, the finance, the legal services, our courts. They love that, but they will have no choice but to go, they have told me."
Finance chiefs had privately told him they were "worried" about the prospect of the two-year Article 50 negotiations resulting in "hard Brexit".
He said: "They have got plans for the possibility of a deal not being done in two years.
"Those with a presence in Europe say it takes between a year and 18 months to get up and go. Others without a presence in a European city, it would take two years."
And he warned that some institutions would not move to European cities such as Frankfurt, Paris and Dublin, which are actively wooing them, but to international financial bases outside the EU, such as New York, Hong Kong, Singapore or Dubai.
Mr Khan, who backed Remain in the 2016 referendum, acknowledged that fears of an immediate flight of jobs from London after the Brexit vote had not materialised.
But he noted that banks, including UBS and JP Morgan, had publicly discussed transferring thousands of staff out of the UK, while HSBC had said that it could move to Paris.
He stood by previous warnings of "colossal damage" to the capital if the UK left the EU without a deal and was forced to fall back on WTO rules, as Prime Minister Theresa May has said she would do if she was offered a "bad" deal by Brussels.
"In most circumstances, no deal means WTO terms, which means tariffs for goods, non-tariff barriers on services," said Mr Khan.
"When I speak to the service sector in particular, no deal and WTO terms equals catastrophe as far as they are concerned."
Mr Khan called on Mrs May to declare on the day she triggered Article 50 that she was offering "a cast-iron guarantee to all the EU citizens who are already here to streamline their permanent residency".
There was already evidence of some of London's one million EU nationals returning to their home countries because of uncertainty over their post-Brexit status, he said, as he warned of the damage which cuts in immigration would inflict on the capital's economy.
"Not only do we need talented people and need immigration but London voted that we want it as well," he said.
"If we can't continue to attract talent it will have a large impact on our ability to be a wealth generator for our country."
Additional reporting by Press Association