Grenfell survivors and relatives of the bereaved have accused the government of trying to demolish the tower block without consulting them.
Members of the Grenfell United group say they have had "previous assurances the tower poses no risk to the community around it".
But a report in The Times claims the government will soon announce the demolition of the block after structural experts advised them it should be "carefully taken down" due to surrounding safety concerns.
It has been more than four years since a fire broke out in one of the building's upper floors, killing 72 people.
Nabil Choucair lost his mother, brother-in-law, sister and their three children in the blaze.
He told Sky News he is "disgusted and disappointed" at the idea of the tower being taken down before a memorial is in place.
"Yet again we haven't been consulted thoroughly and properly. It gets brought out in the media before we get to know about it," he said.
Asked what the tower block represents, he replied: "It represents our innocent families on the day they should have never lost their lives."
Grenfell United, which represents the communities affected by the tragedy, says it is in "shock" over plans for potential demolition, adding that "fewer than 10" of them have been consulted.
They say they were "promised by the government that no decision would be made on the future of the tower without full consultation with the bereaved survivors".
In a statement on Twitter, they wrote: "We struggle to understand why this would be pushed through so quickly.
"It seems to us that removing Grenfell from the skyline while the inquiry and police investigation still continues to only serve those accused or those who haven't acted."
When it was no longer declared a crime scene in 2018, responsibility for the tower was handed to the Ministry of Housing, Communities and Local Government (MHCLG).
A public inquiry into the disaster is still ongoing, which has triggered urgent warnings to remove similar-style cladding from other buildings around the country.
The first phase of the inquiry ended in late 2018, with the second phase set to begin on Tuesday, focusing on how the tower block came to be in a condition that allowed the fire to spread.
The government has said "no decision has been taken".
A MHCLG spokesperson said in a statement: "We know how important and sensitive this decision is.
"Following important independent safety advice from structural engineers, we are engaging closely with the community as we consider the evidence including the safety concerns raised, and what the future of the Grenfell Tower should be.
"We have now published this advice to ensure those most affected have access to the information that will inform a decision on the tower, before one is reached."