Intelligence agencies most have access to the messages and social media sites must do more to police extremist material, she said.
The call comes amid reports that Westminster terrorist Khalid Masood used WhatsApp seconds before launching Wednesday's attack, but agencies are unable to see what was communicated.
Ms Rudd also insisted the likes of Google, which runs the social video sharing platform YouTube, and other smaller sites such as WordPress must realise that they are now publishing - rather than technology - companies and take more responsibility for taking down extreme material.
The Home Secretary left the door open to changing the law if necessary.
But she said she would rather see an industry-wide board doing it independently, as the best people to take action are those who understand the technology and the "necessary hashtags".
On encrypted messaging services, she told BBC One's Andrew Marr Show: "It is completely unacceptable, there should be no place for terrorists to hide.
"We need to make sure that organisations like WhatsApp, and there are plenty of others like that, don't provide a secret place for terrorists to communicate with each other.
"It used to be that people would steam-open envelopes or just listen in on phones when they wanted to find out what people were doing, legally, through warrantry.
"But on this situation we need to make sure that our intelligence services have the ability to get into situations like encrypted WhatsApp."
Ms Rudd said she was calling in a "fairly long list" of relevant organisations for a meeting on the issue this week, including social media platforms.
"What these companies have to realise is that they are now publishing companies, they are not technology companies, they are platforms and we need to make sure that that (hosting extremist material) stops," she said.
"You are right, we will not resile from taking action if we need to do so."
But she went on: "I would rather get a situation where we get all these people around the table agreeing to do it.
"I know it sounds a bit like we're stepping away from legislation but we're not.
"What I'm saying is the best people who understand the technology, who understand the necessary hashtags to stop this stuff even being put up, not just taking it down, but stopping it being put up in the first place, are going to be them."
As well as murdered police officer Keith Palmer, Khalid Masood's victims in the attack included US tourist Kurt Cochran and his wife Melissa, from Utah, who were on the last day of a trip celebrating their 25th wedding anniversary. Mr Cochran was killed and Mrs Cochran was badly injured.
Aysha Frade, who worked in administration at independent sixth-form school DLD College London, in Westminster, also died. She is believed to be a 43 year-old married mother of two.
Leslie Rhodes, a retired window cleaner from Clapham, south London, died from injuries sustained in the attack. He was described by neighbours as a "lovely man".