It is understood that the group moved to the park on Thursday and Richmond Council are now having to take legal action to move them on.
The council has warned residents of Kew Green - where houses are valued at an average of £2 million - it may take a while yet.
On Friday, Richmond Council confirmed it would begin the legal process to obtain and enforce a Possession Order.
Delays are expected due to a High Court order earlier this month that blocked the council’s request to extend the previous expedited injunction process, as is the case with numerous other local authorities.
Instead, the council is forced to go via the courts to remove the group.
A spokesperson also confirmed that once the travellers have been removed it will work to repair any damage caused.
After officers visited the site, some of the group reportedly dismantled posts which had been installed in the ground and packed up their caravans.
Young children played outside and rode bikes, while mothers walked their babies around the park in prams.