Events are due to start on July 23 but, in the latest blow to the organising committee, some 31 towns out of 500 have pulled out of plans to host training camps.
It comes as a fierce publish backlash continues against the Olympics gathers pace because due to spiralling coronavirus cases, The Times reports.
It is feared that Covid outbreaks among visiting teams could overwhelm local health resources.
Japan recently added three more areas to a state of emergency now covering Tokyo, Osaka and four other districts amid surging cases.
Urayasu was due to host 35 British Paralympians and officials in mid-August, but they will now have to travel directly to the athletes’ village in central Tokyo.
Other towns have withdrawn their invitations to the Russian fencing team and the Indian hockey team, as well as to delegations including Belize and East Timor.
It is understood that some delegations are choosing to arrive later for the events, to limit the time spent in Japan, including Britain's Paralympic wheelchair basketball team and America's track and field team.
Meanwhile, a petition calling for the cancellation of the Tokyo Olympics which has 350,000 signatures in nine days was submitted to organisers on Friday.
Stop Tokyo Olympics campaign organiser Kenji Utsunomiya said the global festival of sport - already postponed from 2020 due to the coronavirus pandemic - should take place only when Japan can welcome visitors and athletes wholeheartedly.
“We are not in that situation and therefore the Games should be cancelled,” he told a news conference. “Precious medical resources would need to be diverted to the Olympics if it's held.”
The petition was submitted to the Olympic and Paralympic committee chiefs as well as Tokyo Governor Yuriko Koike.
Asked about the anti-Games campaign, Tokyo Governor Koike said she would continue to work towards a “safe and secure” Olympics.
“Though there is a global pandemic, it is important to hold safe and secure Tokyo Games,” she told a press conference.
The new areas under the state of emergency include the Hokkaido prefecture which reported a record high of 712 case on Thurday.
The region is due to host the Olympic marathon.
Opposition to the Games has also come from doctors and high-profile Japanese athletes including Masters golf champion Hideki Matsuyama and top women's tennis player Naomi Osaka.
Additional reporting by Reuters news agency.