The extraordinary pact sees the party’s MPs sitting alongside the “heirs of Mussolini” in Italy and an Estonian party that celebrates its wartime collaboration with Hitler, among other far-right groups in the Council of Europe.
Yet, Lord Balfe reveals, his repeated attempts to persuade the prime minister to take action have been rebuffed – as he was told it was going on “a long way away and no-one understands it”.
Instead, the peer was kicked out of the European Conservatives Group for protesting to No 10 after the group “invited in” the far-right so-called Democratic Alliance, he said.
“It’s appalling,” Lord Balfe told The Independent, “this group contains virtually every video nasty in the book and we should not be mixed up with them.”
Former Conservative Cabinet ministers and anti-fascist campaigners are urging the prime minister to step in and sever his links with the extremist parties.
Stephen Dorrell, a former health secretary and now chairman of the European Movement, said: “That the prime minister should think this is acceptable is a sad reflection of modern Tory politics.
“To be making common cause with people whose values earlier generations of Conservatives took up arms against to resist.”
David Lidington, the former deputy prime minister and now chair of the Conservative Group for Europe, said “odd alliances” were sometimes formed in the Council of Europe.
But he added: “Having said that, I am deeply uncomfortable about the Conservative Party being associated with some of the political parties in this group.”
The agreement has boosted the power of Tory MPs in the Council, a Strasbourg-based body that pre-dates the EU and pushes for human rights and democracy across the continent.
The size of the 85-strong European Conservatives Group and Democratic Alliance (EC-DA) means it is in line for plum roles such as a rapporteur to troublespots, or to chair key committees.
The Brothers of Italy was born out of the country’s fascist movement, while the first leader of the Freedom Party of Austria – once headed by the notorious Jorg Haider – was a Nazi Minister and SS officer.
Lord Balfe said he has seen the Sweden Democrats “walk around Stockholm with swastikas”, while Bulgaria’s United Patriots “were elected on the basis that there were too many mosques”.
The Conservative People's Party of Estonia attends an annual march to commemorate the Estonian Legion – a unit of the Waffen-SS during the Second World War.
The anti-Nazi Simon Wiesenthal Centre has called the march a “serious concern” because “such ideas paved the way for extensive collaboration by the locals in the implementation of the Final Solution”.
One Sweden Democrat MP, Markus Wiechel, was exposed in 2013 for having referred to black people as “monkeys” and “retarded pension-rescuers” – criticising the idea of immigration to fund an aging population.
The anti-fascist organisation Hope not Hate said Mr Johnson must “intervene immediately” and stop working with “extreme parties with a track record of racism and discrimination”.
“This group includes representatives from the Brothers of Italy, described by some as the heirs to Italian fascism. And for Tory politicians to have any association with figures like Markus Wiechel is a total disgrace.”” said Dr Joe Mulhall, its senior researcher.
Lord Balfe said he met several times with government whips, but was told to drop the controversy – on one occasion, because he “can’t expect the PM to be on top of everything”.
But he added: “What would we say if the Labour party said ‘Let’s have the Communists in’?”
Ian Liddell-Grainger, the Tory MP for Bridgwater and West Somerset, is the chairman of the EC-DA grouping and is believed to have pushed for the pact with the Democratic Alliance.