Downing Street says Jeremy Corbyn is "running scared" of a Brexit debate with Theresa May.
The debate is due to happen on 9 December - two days before MPs vote on the deal Mrs May has secured with the EU - but there is still disagreement over the format.
Mr Corbyn has said he will go with the Tory preference for a BBC debate if it is a straight head-to-head after previously indicating a preference for a similar ITV proposal.
But the BBC offer also involves leaders taking questions from a wider panel and a Number 10 spokesman said they are sticking with that plan.
"A week ago, the PM challenged Jeremy Corbyn to a head-to-head debate. He accepted," the spokesman said.
"Since then, in order to accommodate his confected demands, we've moved our preferred day, accommodated the addition of social media questions at Labour's request, and agreed there should be maximum head-to-head time, while still including voices from employers and civil society in the debate.
"But if Jeremy Corbyn doesn't agree to what's now on the table - a debate on prime time with the prime minister - the public will rightly conclude he's running scared. So let's get on with it."
Labour, however, says it is the PM who is trying to avoid confrontation.
"As she did during the general election campaign, Theresa May is running away from the scrutiny of a real head-to-head debate with Jeremy Corbyn," a spokesman said.
"Why else would she not accept ITV's offer of a straightforward head-to-head debate, as Jeremy has done?
"Instead, her team are playing games and prefer the BBC's offer, which would provide less debating time and risk a confusing mish-mash for the viewing public."
Liberal Democrat leader Sir Vince Cable believes the debate proposals exclude other viewpoints.
"The principal alternative to the Withdrawal Agreement is for the UK to remain as a full and influential member of the European Union," he said in a letter to Sky, BBC and ITV.
Sky News has been campaigning for leaders debates to become a regular fixture of election time and a petition supporting the move has more than 106,000 votes.
Meanwhile, The Daily Telegraph reports that Boris Johnson, Dominic Raab and other former cabinet ministers have written to BBC chairman Sir David Clementi to complain the views of the 17.4 million people who voted for Brexit would be "nowhere represented" in the debate.
They said that a senior Brexiteer should be included in the main line-up and not just on a proposed panel.
Liberal Democrat leader Sir Vince Cable also opposes the proposal for the discussion on the basis that Remainers were being excluded.
In a letter to Sky, the BBC and ITV, Sir Vince put himself forward as an advocate for a so-called people's vote in a second referendum.