The landslide took place on Saturday morning in Atami, southwest of Tokyo, following days of heavy rainfall.
The bodies of what appear to be two victims have been discovered, according to Kyodo news agency.
Shizuoka’s governor Heita Kawakatsu appeared to confirm the deaths, adding: “Because of the heavy rain, the ground loosened and the mudslide occurred… it picked up speed and swept away houses together with people.”
A search-and-rescue effort is being carried out by police, fire and military personnel in and around the town.
A Shizuoka prefecture disaster management official told AFP the local government has requested military assistance for a rescue mission and that “the safety of 19 people is unknown”.
He added that some people in the area are believed to have evacuated but no further details were immediately known.
Television footage showed a mudslide carrying black water and debris slide down a mountainside, knocking down and crushing houses in its way.
One person told the public broadcast NHK: “I heard a horrible sound and saw a mudslide flowing downwards as rescue workers were urging people to evacuate. So I ran to higher ground.
“When I returned, houses and cars that were in front of the temple were gone.”
Heavy rain has been lashing across Japan since earlier this week. Residents in the Shizuoka, Kanagawa and Chiba prefectures have been ordered to evacuate.
More than 2,800 homes in the region were left without power, according to the Tokyo Electric Power Company.
The country’s prime minister, Yoshihide Suga, has put together a task force to respond to the disaster and wider emergency caused by the weather.