Mark Wahlberg Recalls Leaving 2017 Super Bowl After Son Brendan, Then 8, Was 'Screaming and Cursing' at Score

"We were embarrassed," the father of four said with a laugh, recalling the awkward moment in public

Mark Wahlberg is a good sport about his kids' constant jabs.

The actor joked about how his four kids — Grace, 13, Brendan, 15, and Michael, 17, and Ella, 20 — "always make fun" of him while appearing on Wednesday's episode of The Late Show with Stephen Colbert. He then opened up about how he once took Brendan with him to the Super Bowl, though the father-son duo left at halftime.

"I went to do some stuff to promote the movie and then I promised my wife I would be home," he explained. "I took my son and I was a little worried because my son, the last time we went to a Super Bowl, it was the Patriots versus the Falcons."

He continued, "And people always wonder why I left at halftime, but he was 8 years old and he was screaming and cursing at the top of his lungs because they were down 28-3 and we were embarrassed. So I didn't know what he was going to do at the game."

Wahlberg noted the family lives "literally 15 minutes away from the stadium" and was able to watch the game with "no stress."

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

Related: Mark Wahlberg and Wife Rhea Wish Daughter Grace a Happy 14th Birthday with Sweet Photos: 'Keep Shining'

When Colbert asked Wahlberg if it's hard for him and his wife Rhea Durham to see their kids dating, he said, "It's the worst."

"My 18-year-old has his girlfriend and they were cuddled up on the couch watching the Super Bowl. My wife didn't like that at all. She had no problem with the boyfriends coming and going."

Wahlberg clarified that they do like his son's girlfriend, adding, "We already consider her part of the family."

"But she goes upstairs, the door shuts, and my wife is like, 'Open the door, go in there,' " which he objected to. "I'm not going in there. It's not my place, and nothing's happening."

In December, Wahlberg shared a relatable parenting moment at the Las Vegas premiere of his movie The Family Plan.

“I've got a teenager who won't stop playing the video games. I've got a daughter who thinks she knows everything and [me and wife] are complete idiots in many, many ways,” he told PEOPLE.

For more People news, make sure to sign up for our newsletter!

Read the original article on People.