The front of the branch was torn away, with rubble strewn across the pavement and shelves of fizzy drinks visible from the street.
Cambridgeshire Police said officers were called to the Post Office in Kirkgate Street just before 1am on Wednesday.
The suspects had already left the scene with an ATM.
At 4.18am the force was called to another cash machine theft in progress at a Co-op store 27 miles away in Yaxley, near Peterborough.
The suspects rammed a police car as they made off and an officer received “minor injuries”, police said.
The force said the two incidents are believed to be linked to each other, and to two other ATM thefts from last week at Nisa in Chatteris and a Co-op store in Cambridge.
A Post Office spokesperson said it “will be working closely with the Postmaster” of the Old Walsoken branch “to restore service as soon as possible”.
They added: “This type of crime is very traumatic for the retailer and we will be supporting the police in their efforts.”
Anyone with information is asked to call Cambridgeshire Police on 101.