Dr Lee's defection to the party shocked Westminster a fortnight ago when he crossed the floor of the House of Commons while Prime Minister Boris Johnson was at the Despatch Box.
The Bracknell MP's move dramatically robbed the Conservative Party leader of his makeshift majority - a situation the PM worsened when he sacked 21 rebels later that same evening.
But, the former Justice Minister received a lukewarm welcome from members of the Lib Dems due to his past record voting against LGBT rights.
Dr Lee refused to vote in favour of same-sex marriage and joined a separate campaign to bar people with HIV from being able to come to the UK have, causing dismay in the LGBT section of the Lib Dems.
During a question and answer session on Sunday at the party's Bournemouth conference, Ms Swinson was shouted down as she explained why Dr Lee had been accepted.
Councillor Catherine Finnecy shouted out from her seat in the hall that the East Dunbartonshire MP was supporting a "Ukip policy" - a reference to Dr Lee's tabled amendment to the Immigration Bill in 2014 that would have seen immigrants tested for "prescribed pathogens", including HIV, before getting immigration permission.
Speaking afterwards, Ms Finnecy called the decision to allow the former Tory minister to join the party "a massive f***-up".
"She stood up there defending a Ukip policy and claimed it was liberal," said the Chelmsford representative.
"That is against all the advice from the relevant NGOs, including the Terence Higgins Trust (HIV charity).
"Jo and Alistair [Carmichael, chief whip] have been defending it to everyone but it is highly offensive."
The party leader, who took questions for more than an hour, was asked about "how carefully you're vetting" new entrants by a worried member of 27 years.
The questioner said to the leader: "I'm concerned that some of the people who have joined recently don't share those values and that their voting record shows their true colours."
Ms Swinson defended the decision in her reply, saying: "I am confident that all of those people who have joined our party in Parliament do share our liberal values.
"They might not share our views on every single issue but around this hall, we don't share views on every single issue."
But Ms Finnecy, who was elected in Chelmsford in May with the Lib Dem surge ridding the local Tories of their majority, said having an additional MP in Parliament was not worth the fall-out it had created.
Jennie Rigg, who chaired the Lib Dem LGBT caucus, quit the Lib Dems upon hearing Dr Lee had joined her party this month, describing Mr Lee as "a homophobe, a xenophobe, and someone who thinks people should be barred from the country if they are ill".
Ms Finnecy said of Dr Lee's dramatic allegiance switch: "It wasn't a coup - it was over in five minutes.
"He will lose his seat at the election and we will waste loads of resources in attempting to hold it.
"This was an egotistic move and a massive f***-up. This was the straw that broke the camel's back for many of us in the LGBT community. We don't feel listened to."
The Lib Dem member since 2016 said she submitted a question to Ms Swinson asking about the rationale behind Dr Lee's admittance, but it was not chosen.
"It was all a PR exercise," she added.
Mr Carmichael looked to address concerns from the stage on Saturday and vowed to improve the process in future.
Referring to Dr Lee's approval process, he said: "Could we have handled it differently? Could we have handled it better? Yeah, hands up, of course we absolutely could.
"We have not been deaf to the concerns of LGBT+," he added.