Disaster management officials in Shizuoka prefecture in central Japan say the mudslide occurred on Saturday morning in Atami, a town known for its hot springs.
Prefectural disaster management official Takamichi Sugiyama said rescue workers are searching for at least 19 people, adding the mudslide occurred in an area called Izusan.
Mr Sugiyama said some people in the area are believed to have evacuated but no further details were immediately known.
Television footage showed a powerful, black mudslide slide down a mountainside, knocking down and crushing houses in its way as helpless neighbours watched in horror, some recording on their phones.
Heavy rain has been lashing across Japan since earlier this week, causing rivers to swell in central and the Tokyo region, and increasing the risk of landslides.
Defence forces will join firefighters and police in the rescue operation, Mr Sugiyama added. Evacuation warnings have also been issued for a widespread area.
NHK TV footage later showed a part of a bridge had collapsed.
Atami is a quaint seaside resort area in Shizuoka prefecture, about 60 miles south-west of Tokyo. The area that was hit by the mudslide, Izusan, includes hot springs, residential areas, shopping streets and a famous shrine.