There has been unease with the depictions of black women and slavery references in the song, which reached number one in the US in 1971.
Guitarist Keith Richards told the LA Times of the decision but said he was confused by people who wanted to “bury” the track and he wondered where the “beef” was.
“Didn’t they understand this was a song about the horrors of slavery?” he said.
“But they’re trying to bury it. At the moment I don’t want to get into conflicts with all of this s***.”
The 77-year-old musician concluded that he’s “hoping that we’ll be able to resurrect the babe in her glory somewhere along the track”.
The 1971 hit is widely considered to have one of the greatest guitar riffs in history, but its success is thought to have overshadowed the song’s references to slavery, sex and drugs.
Critics have said the track contains “stunningly crude and offensive lyrics” that are “gross, sexist, and stunningly offensive towards black women”.
The 1969 song has been a staple of the Rolling Stones live shows since it came out 50 years ago, and is the second most played song in their catalogue after Jumpin’ Jack Flash, with 1,136 known performances, according to Rolling Stone magazine.
Jagger, when asked about the song’s absence from their recent set lists, simply said they had decided to give the song a break.
“We’ve played ‘Brown Sugar’ every night since 1970, so sometimes you think, We’ll take that one out for now and see how it goes,’” he said.
Jagger explained in an interview in 1995 that he was uncomfortable with the lyrics of the song, which was named as number 495 in Rolling Stone magazine’s list of the 500 Greatest Songs of All Time.
“God knows what I’m on about in that song,’ said Jagger, in the interview. “It’s such a mishmash. All the nasty subjects in one go.”