The MasterChef presenter announced in March 2023 he was leaving the BBC series about how household products are made after eight years and it was reported there had been complaints about "inappropriate" comments he was alleged to have made during filming.
Wallace, 58, told Good Morning Britain: "Offending people in the factory? No. There was an issue, that was not the issue.
"I don't want to discuss the issue. It was a decent time to come out."
When he announced his departure the former Strictly Come Dancing contestant said he was stepping down to spend more time with his four-year-old son Sid.
"Mum needed help."
He insisted the reports about him making "offensive" comments to factory workers were "inaccurate".
Wallace added that the undisclosed "issues" that arose around the series coincided with his family needing him to be at home more, and so they prompted his decision to leave the show.
The TV presenter has previously revealed that Sid was diagnosed with autism in 2022.
Wallace married Sid's mother Anna Sterpini, 37, in 2016.
The former greengrocer met Anne-Marie on Twitter in 2013. They welcomed baby Sid in 2019 after struggling to conceive.
The MasterChef judge and self-confessed pudding-lover used to weigh over 16 stone but went on a health kick in 2017, cutting back on alcohol, fry-ups and takeaways as well as joining a gym, and has lost over four stone.
He was previously married to first wife Christine for just months in 1991. He was married second wife Denise from 1999 to 2004 and he tied the knot with third wife Heidi in 2010. They split in 2012.
Watch: Gregg Wallace discusses son's autism diagnosis