The 'Leave Right Now' singer found fame in 2002 when he beat Gareth Gates to win Pop Idol, which many see as having paved the way for future TV talent programmes.
Young told the PA news agency that Pop Idol was "so pure".
But he said that the way the format has evolved since then makes him “really sad and actually quite angry”.
“I feel a bit unsure as to why a show like X Factor hasn’t had its comeuppance in terms of how it has treated people,” he said of the show, which is being rested for at least five years.
“And I am surprised. There is obviously stuff going on in the background, but hopefully one day it will come out."
The singer, 42, added: "I wish it was my story to tell but it’s not, unfortunately."
Discussing Pop Idol, which was a huge hit in 2002, the star said: “We were very lucky and I think one of the best things is it was so pure.
"It was a singing competition and the public were never duped. They paid money by phoning in. They were the record company and they voted for me to have a record contract.
“They took the power away from the fat cats at the record companies and I will never forget that. They were the ones that gave me a record deal.”
Young – who will celebrate his 20 years in the industry with a tour next year - made the headlines when he was a contestant after he stood up to TV and music mogul Simon Cowell, who was also a judge on The X Factor.
The sharp-tongued judge said Young’s performance was “distinctly average”, but the hopeful told him calmly: “I don’t think you could ever call that average.”
The singer told PA that the exchange was his “defining moment”.
“I have to remind myself how young I was and how powerful he was – and I stood up for myself and for others,” said Young, who was 22 at the time.
“And I am very proud of that.”
This summer it was reported that The X Factor is being rested for at least five years following a fall in ratings.
ITV said at the time that there were “no current plans” for another series of the singing competition.
Yahoo has contacted ITV for comment.
Watch: Little Mix didn't read X Factor contracts before signing