Braverman was told her rhetoric about migrants was similar to that used by the Nazis.
A video published over the weekend shows 83-year-old Joan Salter taking the cabinet minister to task during a meeting in her Fareham constituency on Friday evening.
Braverman thanked Salter for her question but said she “won’t apologise” for her choice of words.
The Home Office has demanded Freedom From Torture, which posted the video, remove it on the grounds it “misrepresents the interaction”.
But the charity has refused to do so and said the home secretary should be “ashamed” of her “dehumanising” comments.
The prime minister’s spokesperson said on Monday: “You’ll know the UK’s record on providing a safe haven to tens of thousands of people, whether it’s people from Afghanistan or other countries and we continue to be proud of that record.”
Asked twice if the PM agreed with Braverman’s language, the spokesperson said: “The Home Office put out a statement on this. I don’t have anything to add to that.”
Braverman initially used the word “invasion” in the Commons in October during a debate on the number of small boats cross the English Channel from France.
Salter told Braverman at the meeting on Friday: “I am a child survivor of the Holocaust. In 1943 I was forced to flee my birthplace in Belgium and went across war-torn Europe and dangerous seas until I finally was able to come to the UK in 1947.
“Now, when I hear you using words against refugees like ‘swarms’ and ‘invasion’ I am reminded of the language used to dehumanise and justify the murder of my family and millions of others.”
She added: “Why do you find the need to use that kind of language?”
Responding, Braverman said her own father had been “kicked out of Kenya” and found refuge in the UK, while her mother came to Britain from Mauritius when she 18 to work as a nurse.
She said: “They owe everything to this country and they have taught me a deep and profound love of Britain and the British people - their tolerance, their generosity, their decency, their fair play.
“And that also means that we mustn’t shy away from saying there is a problem and there is a huge problem that we have right now when it comes to illegal migration, the scale of which we have not known before.
“And I won’t apologise for the language that I’ve used to demonstrate the scale of the problem. I see my job as being honest with the British people and honest for the British people.
“I’m not going to shy away from difficult truths, nor am I going to conceal what is the reality that we are all watching.”
A Home Office spokesman said: “Footage of a conversation with a Holocaust survivor is circulating online. The video has been heavily edited and doesn’t reflect the full exchange.
“Since the footage misrepresents the interaction about a sensitive area of policy, we have asked the organisation who posted the video to take it down.”