The outburst on Britain Decides: Everything But Brexit Debate came after Ms Swinson said her party was being "honest" in announcing it would raise the basic rate of income tax to better fund the NHS if it won Thursday's General Election.
The woman in the audience, whose name was not given, said: "Why doesn't Jo Swinson show her true colours? They're blue.
"You're a Tory in disguise, state what you are. You dare to try and raise taxes on lower earners when the country is in austerity. How dare you."
Ms Swinson - who was part of the Tory and Lib Dem coalition government between 2010 and 2015 - replied to her critic by saying: "And we need to end that (austerity) by investing in our public services and we need to be honest about how we do that.
"If you look at the independent Institute for Fiscal Studies, they have looked at the Labour plan and they say they are not credible."
The comeback saw shadow education secretary Angela Rayner launch into an angry tirade, yelling at the former coalition minister: "You voted for the bedroom tax, you voted for austerity, you voted for tuition fees.
"You did that nine years ago - apologise for tuition fees."
— Rachael Swindon (@Rachael_Swindon)December 8, 2019
The Lib Dem leader responded by accusing Labour of "making a list of promises that you cannot deliver".
The leader of five months said she had apologised for her party's part in hiking tuition fees, from £3,000 per year to a maximum of £9,000 in 2010.
She told the debate audience that she wanted Father Christmas to give shadow chancellor John McDonnell an abacus because "Labour's plans do not add up".
Firing back, Ms Rayner accused the Lib Dems of the same, arguing that there were not enough professional minders to fulfil the manifesto pledge of providing free childcare to all children over the age of nine months.
"You've said you're going to provide early years (childcare) but the truth is the figures don't add up," said the shadow cabinet member.
As well as Labour and the Lib Dems, the SNP, Green Party and Plaid Cymru also took part in the 6.30pm debate.
Channel 4 stood two empty podiums where the Conservative and Brexit Party candidates would have been after both parties decided not to send representatives.
It followed a similar move last month, with the publicly owned broadcaster opted to have two melting ice-sculptures on stage during a leaders climate change debate to represent the absent Boris Johnson and Nigel Farage.
On Sunday, presenter Cathy Newman read out the Tory responses to some of the non-Brexit related questions, but the governing party's decision to skip the debate was heavily criticised during the hour-and-a-half programme.
Ms Swinson said: "You're doing a sterling job, Cathy, of reading out what they would have said but to be honest I think they should have turned up if they wanted to be represented - just like Boris Johnson couldn't be bothered to turn up to the Channel 4 climate change debate.
"It tells you exactly what the Tories think about the importance of the environment in this election."