In a clip published by the Yorkshire Post, Mr Williamson told a meeting hosted by grassroots group Momentum that the party had “given too much ground”.
Mr Williamson also said he had celebrated the resignation of former Labour MP Joan Ryan, who defected to join the Independent Group of MPs last week.
Following condemnation from MPs in his own party, Mr Willliamson has now apologised for his “recent choices of words”.
Labour are also now investigating Mr Williamson but he has not been suspended – something Prime Minister Theresa May said should happen during PMQs.
Jewish MP Luciana Berger, who quit Labour over its handling of anti-Semitism, wrote on Twitter of the video: “This is what I have left behind. It’s toxic. Our country deserves so much better. #ChangePolitics”.
Labour deputy leader Tom Watson described Mr Williamson’s behaviour as “deliberately inflammatory” and is reportedly set to call for his suspension from the party.
Meanwhile, Cardiff South MP Stephen Doughty tweeted: “This conduct is unacceptable and has no place in our party.
“I have made clear to both the leader’s office and our whips that I expect urgent action to be taken.”
Birmingham Yardley MP Jess Phillips said Mr Williamson’s comment seemed “specifically designed to upset”, adding: “Must be disciplined.”
Walthamstow MP Stella Creasy said anti-Semtitism has to be “expunged not ignored”, tweeting: “Those who perpetuate it or minimise it bring the party into disrepute. No ifs, no buts. #standupspeakup”.
And Ilford North MP Wes Streeting tweeted: “Stomach-turning. No action will be taken.”
In the video, recorded at a meeting in Sheffield, Mr Williamson said: “The party that has done more to stand up to racism is now being demonised as a racist, bigoted party.
“I have got to say I think our party’s response has been partly responsible for that because in my opinion … we have backed off on too much, we have given too much ground, we have been too apologetic.”
Mr Williamson also said he had sung Celebration by Kool & The Gang after hearing of Ms Ryan’s resignation.
She was one of nine MPs who quit in protest at Mr Corbyn’s leadership last week, with his handling of anti-Semitism in the party among their central grievances.
The video of Mr Williamson emerged as he came under fire for booking a room in Parliament for a screening of a film about anti-Semitism and activist Jackie Walker.
She was suspended by Labour over allegedly anti-Semitic comments in 2016.
Mr Watson said he reported the matter to the chief whip and general secretary “as soon as” he found out about it.
A Labour Party spokesman said Mr Williamson’s actions had been “completely inappropriate” and fell below the standards it expects of MPs.
During a visit to Derbyshire in January, Mr Corbyn said Mr Williamson was a “very good, very effective Labour MP”.
The Labour leader told Derbyshire Live: “He’s a very strong anti-racist campaigner. He is not anti-Semitic in any way.”
Mr Williamson was first elected MP for Derby North in 2010, but was unseated in the 2015 general election.
He ran again in 2017, this time receiving a visit from Mr Corbyn during the general election campaign, and won.
Mr Williamson was later made a shadow fire and emergency services minister.
He left the post by mutual agreement six months later after commenting on policy outside his brief to suggest that council tax should be doubled for better-off homes.