David Davis has been accused of threatening to resign more times this year than he has met with the EU's chief Brexit negotiator.
The Brexit Secretary was mocked for only holding talks with Michel Barnier twice in the last six months, but laughed off the joke.
His Labour shadow delivered the barb after weekend reports Mr Davis was threatening to quit the cabinet.
Crunch talks were held in Number 10 over the terms of the backstop Northern Ireland border proposal.
In his opening questions to Brexit ministers on Thursday, Sir Keir Starmer said: "I have done a quick tally and I think that this year he has threatened to resign more times than he has met Michel Barnier."
He moved on to challenge Mr Davis over confusion about whether MPs would get to vote on the final Brexit deal - and be able to send Theresa May back to Brussels if they are unsatisfied.
Sir Keir quoted lead Tory rebel Dominic Grieve, a former attorney general, who told Sky News on Tuesday: "We are going to get a meaningful vote on both deal and no deal. I have no doubt about it."
He also highlighted the current solicitor general's admission that "I have a problem both constitutionally and politically with a direction given by parliament".
Asked who was correct, Mr Davis said he was "entirely with" the solicitor general. "That follows automatically," he added.
Downing Street is on Thursday preparing an amendment that incorporates some backbenchers' calls for a "meaningful vote" into the government's flagship Brexit bill.
But concerned backbenchers worry how far it will go.
Peers are set to discuss the law change on Monday, before it "ping pongs" back to the Commons on Wednesday.