Horror is back on our screens: and this time it’s personal

In the absence of the big-budget action movies that were intended to hoist us out of our drab lives this autumn, horror is closing in.

Related: Saint Maud review – nursing a nightmare of erotic intimacy | Peter Bradshaw's film of the week

On TV, HBO’s Lovecraft Country has co-opted the monsters conjured by an early 20th-century master of the genre – HP Lovecraft – into a shape-shifting story of racism in mid-century America. In the cinema, the demons are internalised in Saint Maud, the story of a psychotic nurse who attempts to force her spiritual delusions on the dying woman in her care.

Unlike your average zombie or space-invader shockers, both deny us the security of an us v them premise, conjuring a dynamic that implicates us all. Racism might have unleashed the demons in Lovecraft Country, but their violence is all-devouring. Maud, meanwhile, is a pitiable, and even relatable, inversion of the idea of nurses as angels of mercy. She’s a lonely young woman who fatally confuses spiritual ecstasy with its more worldly counterparts.

The familiar trope of the powerless woman is given a different twist in Adrian Shergold’s Cordelia, in which a traumatised victim of the 7/7 London transport bombings is abandoned for the weekend in a shadowy mansion block by her identical twin, only to find herself in a danse macabre with the cellist who lives upstairs. Like Saint Maud, it’s a chamber piece, entirely dependent on the tension between two needy characters.

These dramas colonise a liminal space between personal and collective trauma. We’ve all heard horror stories about rogue carers, while a relentless media has intimately connected us to crimes born of hate, whether terrorist attacks or discriminatory police violence. Once seen, these can never be unseen: they imprint themselves on our memories so that our personal lives become intimately connected with the public horror.

Cordelia spells this out by not spelling it out: we don’t know if the title character was actually on the bombed tube train, or if she is haunted by the fact that she might have been, or even just that other people were. All we know for sure is that the bomb went off in a tunnel beneath her home, leaving her with a terror of subterranean spaces.

As the film unfolded, it struck me that this is precisely the place occupied by the genre itself today. Streaming has brought more of it closer to more of us than ever before. We watch it in our living rooms and in our beds, recognising tropes developed over decades to render the familiar strange and the strange familiar: a phone rings out in the darkness, spiders scuttle across a wall, a woman lies naked in a bath (you could write a thesis on the symbolism of the roll-top bathtub). All are part of a shared symbolism, which leads us underground to a place where terrors we cannot comprehend, or even see, brush against our faces.

One of the finest 20th-century proponents of the genre, short story writer and novelist Shirley Jackson, is herself the subject of a fine little biopic which investigates her relationship with her imagination and her creepy academic husband. Jackson was a semi-recluse who severely doubted her own gifts, and was largely forgotten after her death in 1965 at the age of just 48.

Later writers, from Stephen King to Neil Gaiman and Donna Tartt, have disinterred her reputation, acclaiming her as a key player in a genre that has gone in and out of fashion over generations. For horror is like one of HP Lovecraft’s shoggoth monsters, a “shapeless congeries of protoplasmic bubbles”, which often rears up at times of great change or catastrophe. Jackson’s stories evoked the terrors of cold war America just as surely as the manmade monster of Mary Shelley’s Frankenstein embodied the anxieties of the scientific revolution, or the ghost stories of the 1920s and 30s summoned the dead of the first world war.

Discussing the revival of interest in his mother’s work just after the 2016 US presidential election, Jackson’s son pointed to one of her short stories, saying, “The Possibility of Evil: doesn’t that title say it all?” So what do the genre’s current manifestations say about us and, just as importantly, what do we need from them?

These new stories urge us not to believe what we are told, or what we think we see. They warn of gaslighting and the abuse of power, telling us that nobody is to be trusted, and that even vulnerability can be malign. But they also implicate us, as the best horror has always done, in the possibility of evil.

By forcing us down into our deepest anxieties they create an explosion of relief. At a time when so many communal experiences are denied to us, they recreate the childish pleasure of squealing with terror while holding hands in the dark. So while trick-or-treating may have been cancelled this Halloween, and the big thrills are on hold, fear not: there is always something small and scary ready to crawl, rear or slither out of the screen.

• Claire Armitstead is associate editor, culture for the Guardian