According to Evening Standard writer Rohan Silva, Prince Andrew used the n-word at a Buckingham Palace meeting with him when he was a Downing Street advisor.
Silva, who is of Sri Lankan heritage, says the incident took place during a discussion in 2012 about trade, and came as he asked the duke whether the government department responsible for trade "could be doing a better job".
He claims Andrew replied: "Well, if you pardon the expression, that really is the n***** in the woodpile."
Silva said the exchange left him "reeling".
He went on: "For a long time afterwards I kicked myself for not confronting the prince on his choice of words - and it's something I still regret today. After all, he clearly wasn't taken to task very often by the people around him, which meant offensive language went unchallenged."
The phrase "n***** in the woodpile" originated in the American Deep South in the 19th Century and was used to describe fugitive slaves who hid in piles of logs as they fled north to Canada.
It was once commonly used to refer to a hidden problem, but is now considered unacceptable.