Ms Waters, the founder and director of Sharia Watch UK, was beaten by the lesser known Henry Bolton who is a former soldier and Thames Valley police officer.
Ms Waters, also a co-founder of the UK branch of the anti-Islam group Pegida, was a deeply divisive candidate with an impassioned support base. Her unexpected loss has saved Ukip from a lurch towards the far right and a threatened mass walkout of party members.
The narrow favourite to win, who has described Islam as “evil”, tweeted “Today: Jihad – 1. Truth – 0” after learning of her defeat in the seven-horse contest.
The 40-year-old finished second, with 2,785 votes, a significant stretch behind the 3,874 won by Mr Bolton.
Mr Bolton, who is rumoured to be a former Liberal Democrat, said the right-wing party had dodged becoming the “UK Nazi Party” by electing him as leader.
The new leader used his victory speech at the party’s conference in Torquay to urge members to come together in the face of mounting divisions, saying: “Without being united, we cannot lead.”
Mr Bolton is faced with the mammoth task of rescuing Ukip who gained fewer than 600,000 votes at this year’s general election just two and a half years after it was the UK’s third-biggest party by vote share at the 2015 general election.
The new leader denied he was a caretaker for Nigel Farage and said he would be talking to him in the coming days about what role he could play in the party.
Mr Bolton sought to distance himself from the anti-Islam rhetoric espoused by Ms Waters. “There is an issue to be discussed. I abhor the rhetoric that we are war with Islam,” he said.
Ms Waters, whose campaign video stated “Islamic culture does not fit with ours”, launched the UK branch of Pegida, a German far-right and anti-Islam group, with former English Defence league leader Tommy Robinson in January 2016. Last year, Mr Farage said party members should not associate with Pegida or take part in marches.
Earlier in September, Ms Waters said she would not prevent Robinson, who endorsed her bid for leadership, from joining Ukip. She said that while she did not think he had any intentions of joining the party, she would not be opposed to his membership if he were to express interest.
Mr Farage, who had hinted at launching a new party if Ms Waters were to have clinched it, said he was “delighted” about Mr Bolton’s victory, adding: “He’s a man of real substance.”
Arron Banks, a multimillionaire who was one of Ukip’s biggest donors, indicated he may begin backing Ukip again. The insurance mogul tweeted: “This is great news and I’m sure both myself and Nigel will want to re-engage with the party!”
Ms Waters’ mounting popularity caused a large influx of new members to Ukip, prompting fears of “infiltration” from some party members and a number of MEPs reportedly threatening to quit if she won.
The Dublin-born politician has been explicit about her hopes to shift Ukip to a form of cultural nationalism based on religious intolerance. Her manifesto warned that Islam has turned Britain into a “fearful and censorious society”, called for the burqa to be banned, the closure of all sharia councils and a freeze on all immigration.
The activist’s views echo extreme nationalist groups such as the English Defence League (EDL) and Britain First. She has close links to the far-right and gained support in her campaign from Jack Buckby, a former prominent member of the British National Party who then spent time in the overtly racist group Liberty GB.
Ms Waters, who tried to run as a Labour candidate before joining Ukip, has written for far-right, pro-Trump news site Breitbart. She has penned pieces for them praising Geert Wilders, a populist far-right Dutch politician who wants to close mosques in Holland, and another linking Muslim immigration to rapes and sexual assaults.
She has described Islam as “an expansionist, political, totalitarian and supremacist faith, commanded to world domination” and claimed that people are wary of the religion because they fear their children will be abused.
Under Paul Nuttall, Ukip had already begun to shift towards the right, pledging to ban the full-face veil and outlaw sharia law in its general election manifesto. This marked a significant shift from its 2013 stance of not pursuing a blanket ban on face-covering veils.