The reality competition show aired on ITV from 2004 to 2019, with talks reportedly in place to revive the series once again.
However, the Sunday Mirror now reports that a group, consisting of five solo acts and one band, have approached a major law firm to help sue SyCo.
Speaking to the publication anonymously, one of the acts said that they were going forward with allegations of “bullying, mistreatment and neglect” from their time on the show.
Describing their experience on The X Factor as “horrific”, they said: “Myself and five former acts have agreed to join forces and get the justice we feel we deserve.
“Simon needs to take responsibility. We lost a lot of money from having periods of our lives not making money to deal with trauma still ingrained from the show.”
The group are reportedly looking for compensation after spending time and money “trying to make our lives normal again”.
“We have a lot of evidence, including emails, recordings and psychological assessments that will show how we were mistreated,” they said.
SyCo told The Mirror that Cowell himself was reportedly not aware of, or involved in, any of the allegations.
A spokesperson for The X Factor told The Independent: “The welfare of those taking part in our programmes is our number one priority and we have robust measures and welfare teams in place to ensure support is always available.
“We are unaware of any legal claims being filed against The X Factor and despite not being on air for several years, we always remain open to discuss concerns with anyone who chooses to do so.”
Earlier this month, it was reported that work is currently under way on a documentary set to expose claims of bullying, exploitation, and harassment on the show.