The blaze broke out at 9pm local time on Saturday in Sitakunda, following an explosion in a container full of chemicals.
The first blast then caused a chain reaction across the port as the fire began to spread - 450 people have also been injured.
“So far 40 dead bodies reached the morgue here,” a police officer at the state-run Chattagram Medical College Hospital (CMCH) was quoted as saying by news agency PTI.
The depot is located near the country’s main Chittagong Seaport, 134 miles (216km) south-east of the capital, Dhaka.
According to Main Uddin, director general of the Bangladesh fire service and civil defence, at lease five firefighters have died. A further 15 are being treated for burns, he added.
Although the death toll had reached 38 by Sunday afternoon, the area's civil surgeon said the number could still rise.
“The explosion just threw me some 10 metres from where I was standing. My hands and legs are burnt,” lorry driver Tofael Ahmed told AFP news agency.
Prime Minister Sheikh Hasina expressed her shock at the accident and ordered arrangements for the medical treatment of the injured.
Firefighters were still struggling to extinguish the flames in parts of the port.
The army has been deployed to prevent chemicals flowing into the sea.