Brady was the star of the show back in February as the Pats battled from 25 points down to win a fifth Vince Lombardi trophy in Super Bowl 51 in Houston.
Tradition dictates that the winning team visit the White House and the incumbent president in the following offseason but Trump's controversial time in office has a number of players likely to forego the trip.
Brady, who has sided with Trump on a number of occasions in recent years, will not be present today with the 39-year-old confirming he has some "personal family matters" to attend to instead.
"I am so happy and excited that our team is being honored at the White House today," he said in a statement on Wednesday. "Our team has accomplished something very special that we are all proud of and will be for years to come.
"Thank you to the President for hosting this honorary celebration and for supporting our team for as long as I can remember.
"In light of some recent developments, I am unable to attend today's ceremony, as I am attending to some personal family matters. Hopefully, if we accomplish the goal of winning a championship in the future years, we will back on the South Lawn again soon. Have a great day!"
Safety Devin McCourty is one of those boycotting the visit and explained why in a recent interview with Time magazine.
"The different things that come out of the White House or the [administration] just didn’t agree or align with some of my views,"