Labour MPs are furious about what one called the “muddle and confusion”, which they blame for putting the UK at greater risk of a Boris Johnson government crashing out of the EU in October.
But Dame Margaret, a former foreign secretary and temporary Labour leader, argued the “stumbling block” was not Mr Corbyn himself, but “the leader's office”.
Speaking to BBC Radio 4, Dame Margaret said: “I'm beginning to think that some of them do actually want Britain to leave the EU no matter what.
“They don't give a toss about what the British people now want or what Labour members think is in the country's interests.
“They just are determined to make sure we don't do anything to impede Britain leaving, if necessary with no deal.”
Asked who she meant, Dame Margaret added: “I think there are people very close to him, with great influence with him, who are and have been from the beginning passionately opposed [to EU membership].
“He wants to keep the party together as much as possible and present a united front on the issue.”
Some have referred to “the four Ms” in Mr Corbyn’s team, all known opponents of a fresh Brexit referendum and some of whom have a long antipathy to the EU.
At Tuesday’s shadow cabinet, Mr Corbyn refused to budge on his stance that “any deal” to be “put to a public vote", which could mean a general election or second referendum.
Frontbench supporters of another referendum are desperate for Mr Corbyn to start campaigning for it – and to guarantee Labour would back Remain if it took place.
They were alarmed, last week, when the leader said any referendum ballot paper should contain “real choices for both Leave and Remain voters”.
He also floated the option of copying Harold Wilson’s approach, during the 1975 referendum, when the then-leader took no active position, allowing both wings of his party to fight it out.