Prince Charles Asked About the Skin Color of Meghan Markle and Prince Harry's Baby, New Book Claims

Prince Charles was the unnamed royal who speculated about the complexion of Meghan Markle and Prince Harry's future children, according to a new book.

In Brothers And Wives: Inside The Private Lives of William, Kate, Harry and Meghan, by Christopher Andersen, a source close to the royal family claims that Charles made the comment during a conversation over breakfast with his wife Camilla on the day the Duke and Duchess of Sussex's engagement was announced.

"I wonder what the children will look like?" Charles mused to Camilla, according to Page Six.

Camilla was said to be "taken aback somewhat by the question" and noted that the child would be "absolutely gorgeous."

For more on Prince Charles and other top stories, listen below to our daily podcast on PEOPLE Every Day.

Allegedly lowering his voice, Charles then asked: "I mean, what do you suppose their children's complexion might be?"

A spokesman for Prince Charles' office of Clarence House denied the report, saying: "This is fiction and not worth further comment."

In an interview on Today on Monday, Andersen elaborated on the claims in his book, saying that Charles' comments were "benign" and twisted by palace courtiers.

RICHARD POHLE/POOL/AFP via Getty Images Prince William, Prince Harry, Meghan Markle and Prince Charles

"On the morning that Meghan and Harry's engagement was announced, in a very kind of benign way, Prince Charles started to muse on what their future grandchildren might look like," Andersen told hosts Hoda Kotb and Savannah Guthrie.

"I mean, here's this beautiful biracial American woman and the world's most famous redhead. I'm a grandfather, of course, we all do this, speculate on it. But it was turned into something very toxic, it was weaponized by the 'Men in Gray' who run the palace organization. Unfortunately, by the time it got to Harry, that's the way he took it," he said.

Samir Hussein/Samir Hussein/WireImage Meghan Markle, Prince Harry and Prince Charles

RELATED: Private Investigator Apologizes for 'Robbing' Prince Harry of His 'Normal Teenage Years'

Meghan and Harry, who are parents to 2-year-old son Archie and 5-month-old daughter Lili, referenced the comments in their interview with Oprah Winfrey last March, with the Duchess of Sussex saying there were "concerns and conversations about how dark [Archie's] skin might be when he's born."

Max Mumby/Indigo/Getty Images Prince Charles, Meghan Markle and Prince Harry

An astonished Oprah asked Meghan who made that comment, with Meghan declining to answer, saying the revelation would be too damaging.

Later in the interview, Oprah pressed Harry on the issue, asking him who was behind the racially charged comment.

"That conversation I'm never going to share," he said. "It was awkward. I was a bit shocked."

Harry later told Oprah that it was not his grandparents Queen Elizabeth and Prince Philip.

He also said that the conversation happened early in his romance with Meghan. "That was right at the beginning: What will the kids look like?' "

Samir Hussein/WireImage Prince Charles, Prince William, Kate Middleton, Meghan Markle and Prince Harry

A statement released by Buckingham Palace at the time on behalf of Queen Elizabeth addressed the issues raised during the Sussex's interview.

Can't get enough of PEOPLE's Royals coverage? Sign up for our free Royals newsletter to get the latest updates on Kate Middleton, Meghan Markle and more!

"The issues raised, particularly that of race, are concerning," the statement said. "While some recollections may vary, they are taken very seriously and will be addressed by the family privately.

In the days following Harry and Meghan's interview with Winfrey, Prince William told a reporter, "We are very much not a racist family."