The comedian was interviewing Sir Patrick Stewart on his chat show when he made the remark.
Stewart, 82, had been talking about being knighted by the late Queen in 2010 and at one point jokingly asked Colbert whether he’d want the honour.
The TV star said he wouldn’t say no “if he (King Charles) were just tossing them out” but said they should probably go to “somebody who has done something for the kingdom” before him.
“Because I’m not a huge fan of the royalty, I’m not a huge fan,” he explained.
When Stewart quizzed him about it, Colbert went on: “I just think they are basically gangsters in nice clothes.”
The talk show host recently spent some time with Prince Harry, when he welcomed him onto his programme to talk about the royal’s controversial memoir, Spare.
During the interview, Harry took part in a light-hearted skit that saw Colbert appearing to greet him along with trumpet players in formal dress.
As the Duke noticed his regal welcome, he told the TV star to stop, insisting that all the fanfare was appreciated but that it wasn’t needed.
But Colbert then told him that all the fuss wasn’t for him.
“Get out of the way, he's coming!" he yelled, as he peered around Harry to look past him at somebody else.
Hollywood star Tom Hanks then appeared around the corner with a big smile, saying: "I'm back!"
"Where's my fanfare?" asked the movie star.
Watch: Patrick Stewart on “standby” to return as Professor X