Health Secretary Sajid Javid said he was "very upset" when he saw the video of the PM's adviser joking about a Downing Street Christmas party while London was under Tier 3 restrictions last year.
The cabinet minister told Sky News he pulled out of media rounds on Wednesday because of how upset he was about the video, which Allegra Stratton resigned over yesterday.
He told Sky News: "I didn't appear because I saw that video, it upset me, it upset lots of people, it upset the prime minister.
"I spoke to colleagues in Number 10 and we agreed it was best to have some time to respond to the video, and the prime minister did so, he ordered an investigation by the cabinet secretary."
He added that he did not know why Ms Stratton quit as he called her an "outstanding" spokeswoman who did "fantastic work" on the COP26 climate conference, for which she was a spokeswoman.
Footage showed the former adviser and other aides laughing about a party that allegedly happened on 18 December while people were not allowed to mix with other households indoors as London was under Tier 3 restrictions.
Mr Javid said he has been assured by senior colleagues that no rules were broken but refused to "name names" over who gave that assurance.
He added that Cabinet Secretary Simon Case has the power to "look at any aspect, any potential event, including looking into what happened at the Department of Education" as part of the investigation into the party.
However, the health secretary would not confirm whether that meant looking into other parties said to have taken place around the same time, including gatherings in the PM's own flat and at the Conservative Party's headquarters (CCHQ) on 14 December last year.
A Conservative Party spokesman confirmed on Wednesday evening it was made aware of the party in the basement of CCHQ organised by London mayoral candidate Shaun Bailey's campaign.
"Formal disciplinary action was taken against four CCHQ staff seconded to the campaign," he added.
Jonathan Haslam, former No 10 director of communication for John Major, said the saga is "pretty serious" as it detracts from much more severe issues such as the pandemic and troops on the border in Ukraine.
He said the public will have no patience with the government trying to "wheeze out of it" by trying to call the party a "gathering".
"It's a completely dysfunctional Downing Street, with nobody seeming to have a grip," he told Sky News.
And he added that the public is very tolerant but there will come a point when they are not, which could be a problem as further COVID restrictions are brought in.
On Wednesday evening, the Met Police revealed it will not investigate the 18 December Christmas party due to a lack of evidence and a policy not to investigate retrospective breaches of coronavirus regulations.