Aspire won 24 of the 45 seats in the council, while Labour took 19 seats despite having previously held 40. The result would mark just the fourth time Labour has not held overall control of the borough since its creation in 1964.
Meanwhile, both the Conservative Party and the Greens won one seat each.
It comes after Mr Rahman, a convicted fraudster, made a shock political comeback by taking the mayoralty back from Labour, just seven years after he was removed from office.
Mr Rahman, 56, unseated John Biggs with a combined 40,804 votes, seven years he was found guilty of “corrupt and illegal practices” and banned from running for five years in 2015.
He won the second round having run under his Aspire party with 40,804 votes to Mr Biggs’s 33,487.
Mr Rahman, a former Labour councillor, became the first directly elected mayor of Tower Hamlets when he stood as an independent in 2010.
In 2015 the election court found Mr Rahman guilty of corrupt and illegal practices, and he was banned from running for office for five years.
Tory peer Lord Moylan called the Tower Hamlets result an “absolute tragedy” and said it had been the only seat he wanted Labour to win.
Conservative Tower Hamlets councillor Peter Golds said: “Democracy is democracy - but can a leopard change its spots?”
He added that the Labour candidate had “spent his time foolishly attacking” Mr Rahman.
Mr Rahman was Labour’s choice for mayor in 2010 but was removed over allegations he had signed up fake members, and won as an independent. Last month he said: “I have always maintained my innocence. I believe that Electoral Commission report was a travesty of justice.”