The actors, who play warring brothers Thor and Loki, decided that throwing a real punch would make the scene more believable.
Speaking on The Late Show with Stephen Colbert, Hiddleston said: “There was a scene in Avengers where Thor had to strike Loki across the face and I was wearing the horns, which weigh about 30 pounds.
“I couldn’t really sell the hit, the smack. So I just said to Chris, ‘I think you should just hit me in the face'”.
He added: “It was a terrible idea. I went down like a stone.”
Hiddleston is reprising his role as Loki for an upcoming Disney+ spin off show about the Marvel superhero.
Hemsworth, meanwhile, can next be seen in the Jay & Silent Bob reboot.