A Labour revival must tap into the energy for change on the ground

Ten days before the election, I was in the Dumbiedykes estate in Edinburgh, a high-rise scheme a few minutes’ walk from the Scottish parliament, where a shrinking working-class community is hanging on in the face of gentrification. The neighbourhood, which had a Labour MP until 2010, is the headquarters of an organisation called Edinburgh Helping Hands, whose motto is “Solidarity not charity”: its work runs from giving children in the city free bikes, through football and boxing training, to basic help with the most urgent aspects of poverty.

Its co-founder, Jim Slaven, was readying himself for a food-bank collection at Hibernian FC that would bring in huge amounts of provisions, donated by a constant stream of fans bringing bulging carrier bags. But before he set off, he walked us round his estate, and put its experiences in the context of politics. “Do you know who we never see round here?” he asked. “The left.”

When I worked on films for the Guardian’s Anywhere But Westminster series with my colleague John Domokos during this election, a big part of our plan was to cover the kind of community initiatives that Helping Hands exemplifies. The aim was partly to ensure that the stories we told always contained a large element of hope. But as we went on, we were also struck by the gap between what was happening on the ground and the distant noise of national politics.

A week before our trip to Scotland, we had been in Grimsby, in a seat the Labour party would soon lose to the Tories after 74 years of dominance. We spent an afternoon on the Nunsthorpe estate, at Centre4, a trailblazing project housed in a former secondary school that does everything from funding new businesses to working on loneliness and social isolation. As in many of the projects we visited, most of the volunteers and activists were women, many of whom were so adrift from politics and the rituals of the election as to think that they were almost irrelevant.

One woman we spoke to ran a charitable florist, aimed at bringing people a bit of what she called affordable beauty. “I feel very disillusioned with the parliamentary process,” she said. “I think we need a new party of some kind. I just don’t think they pay any attention to the north.” People in her town, she told us, wanted to see “actual, tangible transformations” and be directly involved in changing where they lived as a matter of personal pride. The problem was that this was something that national politicians seemed barely to understand.

Three days after Labour’s crushing defeat, these encounters feel even more loaded with meaning. Even at the time, they spoke volumes about Labour’s predicament: if even people so driven to help their communities were uninterested in what the party had to say, was it really surprising that it seemed to be losing voters in their droves? But our conversations also highlighted Labour’s lack of roots in many of its supposed heartlands, and its weak relationships with modern successors to the kind of working-class self-help on which the party was built. If Labour is to have any hope of revival, this is where it has to start.

This is complex, nuanced stuff. Plenty of Labour members and activists are involved in such grassroots initiatives, and some of Corbyn’s allies set great store by the idea of “community organising”. But the latter have never really seemed to get beyond the idea that trying to get involved in people’s everyday lives is a good electoral strategy. And besides, Corbynism’s promise of change was always let down by the sense of distant people at the top with no feel for the pulse of the country they wanted to govern.

Many of the reasons for last week’s defeat have already become cliches, though that does not dilute their truth: people’s dismay with Corbyn’s leadership, the effectiveness of the Tory slogan “get Brexit done”, the fact that Labour’s relationship with its old, working-class base has been fraying for decades. But other parts of the story are still in danger of being overlooked. On the national stage, when Corbyn’s Labour tried to speak to the party’s old heartlands, it rarely found a register beyond a condescending narrative about decay and hopelessness, and the idea that only politicians in London could help: part and parcel of Labour’s enduring centralism, and a belief in the unreconstructed state that was ingrained in the old-fashioned hard-left politics Corbyn brought with him.

Late last year, for instance, the party unveiled a party political broadcast and online video titled Our Town. It was acclaimed at the time as a bold step into new political territory, and it was undeniably emotive and affecting – but its narration reduced complex and nuanced realities to melodramatic tropes about abject decline. “These streets were once full of spirit, and hope … years of austerity have ripped the heart out of this place … we lost the factories, we lost the jobs, we lost confidence in our community.” The film turned on the promise that “change is coming”, before cutting to images of the party leader, as if he was the only person who could help. The point Labour missed was that change is already happening, all over the country, and one of its roles should be to encourage and accelerate it. Instead, it offered a cold political giganticism, which might have sounded convincing in the age of the factory and the grand national plan, but was quickly found wanting.

Labour needs to be open to people from beyond its usual circles, keen to learn, and ready to listen to voices it might find discomfiting. In that context, rules and structures are part of what needs to be looked at, but there are far more important questions of culture, and the party’s collective mindset. Many of the people who joined the party in response to Corbyn’s arrival at the top were not the credulous disciples a lot of people might imagine. But nonetheless, the party’s outward image strayed dangerously close to surreal devotion to one man, a tendency to treat any criticism as heresy, and a fondness for political arcana and faction-fighting that were always going to repel far more people than Corbynism attracted.

Related: Labour’s ‘red wall’ is looking shaky. But the problems started decades ago | John Harris

Over the weekend, I put in a call to a Labour person who won. Luke Pollard is the spirited, energetic MP for Plymouth Sutton and Devonport, a rare red dot in an ocean of Tory blue, and part of a working-class city where 60% of voters backed leave. From the outside, people might assume that the presence of thousands of university students in his seat is its defining feature, but he also represents large areas with the same basic features as the places Labour lost. His local party, he says, is “mixed”: there are Momentum members and self-styled moderates, and “a lot of people who don’t really back one side or the other”. He has activists who are involved in youth clubs, and homelessness initiatives. One of his local party’s key assets is the presence of former teachers, who often have a strong connection to many of the people on whose doors they regularly knock.

Crucially, the Labour party in Plymouth takes its place in a web of social enterprises and co-operative organisations, many of which have been fostered by its endlessly innovative Labour council. Pollard has campaigned against defence cuts that would have a dire effect on the local economy, and on the dangers of local tower blocks with Grenfell-type cladding, all publicised via Facebook videos that his constituents often mention without prompting. When last Thursday’s exit poll came through, he feared he was about to lose his seat. But in seven of the eight council wards in his seat, he came top. Somewhere in all that is the beginning of hope, and lessons Labour has to learn as a matter of urgency.

• John Harris is a Guardian columnist