How can Labour remake itself?

The danger now is that people jump to the quick fix, apportion blame and say, “I told you so”. But what was revealed last night, which has been brewing for years, defies anything but the deepest soul-searching about what has gone wrong for Labour. This is existential.

Related: Labour won’t win again until it works out why it lost | Gary Younge

So yes, it was partially the leader’s fault; and yes, Brexit was a horrendously tough issue for Labour to manage, alongside the cultural split between towns and cities, and the toxic antisemitism that’s infected the party. But it’s much more than any of this.

For the crisis is not Labour’s alone. It is a crisis of social democracy that has been gripping the left the world over. So far Labour has bucked the trend, not least because our voting system artificially pumps the party up, as it did in 2005 and 2017. But political gravity cannot be resisted for ever.

Labour now joins the ranks of the French socialists, the German SPD, the Dutch social democrats, the Australian Labor party and so many others in or near the departure lounge of politics. And the test here is not “can you be fleetingly in office?”, but “are you a growing and energetic force capable of addressing the climate crisis and big capital; are you a force for the transformation that society needs?” Unless we understand what’s happened, and how such a force is now created, then any recovery will be built on sand.

Like a leaking pipe, social democracy has been in slow decline for decades. The erosion of class identities, the decline of factory production that produced social solidarity, and then the fall of communism – all pulled the rug from beneath the feet of social democracy. It lost its agency and its method of operation. It lost everything.

Next came globalisation, individualisation and consumerisation. Labour eventually adapted to the terrain in the shape of New Labour, but that sugar rush of repositioning only worked by sowing the deeper seeds of discontent we saw erupt yesterday. Corbynism tried to put a leftwing sheen on a culture of Labourism – the belief that the party has a monopoly of wisdom and therefore knows what is best for the people. Last night that crashed, too.

But throughout this decline something amazing has been happening. The failure of the free market and financialisation in 2008, and the obvious limitations of the old bureaucratic state to create a good society, has opened the space for people to do things themselves. They have had no option.

In communities, neighbourhoods, different sectors and places, people are combining in new and collaborative ways to fix their own problems. This activity in politics, housing, green energy, caring, lending, learning, fixing and making is being accelerated and aggregated by technology that allows us all to know, talk, connect and organise. Today we live with an inalienable right to participation.

So, in this very new context, can Labour come back as it did in 1964 and 1997? In part that begs the question: is there a Harold Wilson, Neil Kinnock or Tony Blair figure who can lead that long march? But the bigger question is, can Labour – a party of technocratic solutions to people’s problems – understand the collaborative sentiment and culture of the 21st century?

In many ways what is needed now resonates with Labour’s foundation. The party came from the world around it – of factories, yes, but of a rich cultural and social mix of clubs and friendly societies. If Labour is to renew itself then it must be immersed in the world around it now, the groups and communities springing up across a range of sectors that try to meet people’s needs collaboratively.

This is a very different role for Labour. It’s one where it would be in service to people and the rising tide of emerging action – not least on climate.

The good society was never going to be created from the top down. It could never be done to us, however well meaning. A genuinely good society can only be created by us, for us. The job of Labour now is to make that happen – or get out of the way and let others do it.

Labour had nothing to say during the election campaign about democracy and power: it was all about what it would do for us. This is the big seismic change the party has to make. It must look to the communities it wants to represent and instead think of them as communities to serve. Then it doesn’t matter if you’re in a town or city, voted leave or remain – you know Labour is there to help you and your family. Then power is something Labour would actively share with others and not monopolise. And a new universal agent is created – the networked citizen.

There is nothing for Labour to go back to. Instead this is the huge cultural leap Labour must make: nothing else really matters.

• Neal Lawson is a director of the centre-left pressure group Compass