B.J. Novak Says Usher Was 'Furious' with Him Over a Rapping Prank on Punk'd

B.J. Novak left Usher going "OMG" after a Punk'd prank gone sour.

During an appearance on The Late Show with Stephen Colbert, The Office alum reminisced about his first-ever TV job on Ashton Kutcher's MTV series. Novak, now 42, said many celebrities he set up on the series got "terribly mad" at him — including Usher.

"I'm meeting all these celebrities for the first time, right? It's thrilling for me. I'm meeting Missy Elliott. I'm meeting Usher," he recalled, adding that his best day was the stars' "worst day of their life."

RELATED: Ashton Kutcher Doesn't Want to Bring Back Punk'd Because 'It's Nice to Have Friends'

On season 2 of Punk'd, Novak met the R&B crooner while posing as a store owner on Melrose Avenue in Los Angeles. At the time, Usher's little brother — who was secretly in on the joke — was being busted for shoplifting.

"The only way I would let the brother go is if [Usher] recorded a rap jingle for my store, which I rapped for him," explained Novak. "And he was like, 'First of all, I'm not a rapper. Second of all, why does it refer to ice?' And I'm like, 'Well, we wanted Vanilla Ice.'"

Novak said Usher, now 43, was less than amused.

MTV UK/YouTube

"He's furious and then Ashton comes out, and he's like, 'Bro!' And like, [gives Usher] a huge hug. And I'm like," the Vengeance director said, opening up his arms in a hug gesture. "And he's like, 'No, no, no, no.' Like, your first impression of someone sticks, you know. So I have not run into Usher since."

"I don't think he'll be in my next movie," Novak joked.

RELATED: Jason Momoa, BJ Novak & 45 More Stars You Won't Believe Are Turning 40 This Year

Mark Sagliocco/WireImage, Marcus Ingram/Getty

Co-created by Kutcher, Punk'd ran for nine seasons on MTV from 2003 to 2015. Countless celebrities were pranked throughout the show's run, including Taylor Swift, Kanye West, Elijah Wood and Salma Hayek.

The That '70s Show alum also recruited future stars to help him pull off his various schemes, including Novak, Dax Shepard and Bill Hader.

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

Earlier this week, Kutcher shared that he's not inclined to revisit the series because "I have friends again."

"For a long time, I would go out and like nobody would want to hang out. They were like, 'I am not messing with him,'" he told Extra on Monday. "It's nice to have friends."