On Victoria's chilly beaches, no one has a tan and excess body hair can only help

Before I moved to Sydney I would look at Max Dupain’s famous Sunbaker photo and think: I do not recognise that beach. Where, for example, is the person in the overcoat and beanie? Where is the person covered in sand? Where are the surfers in full steamers with blue extremities?

If you are from the south-west coast of Victoria, and your ocean is the mighty Southern Ocean, none of the eastern seaboard beach iconography resonates. The south-west coast has some of the most dramatic, beautiful, inhospitable, remote and wild beaches in the world. And much of it is best experienced at a distance.

Growing up in Warrnambool, there were only two hot days a year and even then the water was rough and freezing. Southerly blusters would barrel down the beach. You hardly ever saw anyone sit on the sand – it was like having a mini microdermabrasion.

No one I knew had a tan, unless they had rich parents and could afford to holiday on the Gold Coast. Nor were there any businesses in town that offered waxing. No need. You rugged up to go to the beach. Excess body hair was a good thing!

Life alongside the Southern Ocean was experienced in different ways than life at the more basic beaches of the Pacific.

The novelist and keen surfer Jock Serong, from nearby Port Fairy, puts it this way: “Culturally in the south-west you go to the beach to do something – you fish, you birdwatch, you surf.”

Sitting on your towel, on the sand, and reading a book is a rare sight on the Southern Ocean beaches.

Growing up, the beach was great for walks while contemplating weighty things – the meaning of a confusing Smiths lyric, a complex Yeats poem or the point of life. You could walk for hours along dramatic coastal paths and broad beaches, sand ridged in patterns the wind made, and seldom see another person. You might glimpse the migrating southern right whales coming into Warrnambool to calve or if you were further up the coast towards Portland, colonies of seals. And all along that coastline, the tiny marks in the sand, made by the light feet of the hooded plover.

It even smells different on the south-west coast. It’s the intoxicating scent of salt and the dry grass that grows in tufts from the sand dunes. Years of taking the train home down the coast, and I’d always feel roused when I could start to smell that clear, dry grassy smell. Almost home.

For almost two years I worked in the town of Portland, 100km west of Warrnambool on the far south-west coast. It was a working-class, remote place that happened to have some of the most stunning stretches of coastline in Australia, and that few people visited. Even now there are only two reviews for the marine park on TripAdvisor.

The marine park, Discovery Bay, is 55km of ocean beach about 20km from Portland, towards the South Australian border. The area is rich in bird and marine life – there are high coastal cliffs made of basalt and limestone, freshwater lakes and swamps. Fishermen dive for abalone and pipis.

After work, me and some friends would drive down to beaches, park our cars and walk through the barely-there coastal track, careful of the snakes, and emerge on the other side of the dunes, at the most stunning, wild stretch of beach.

Most days this beach had not a single footprint. It was as empty as the moon. The water whirled and spat in strange, unknowable patterns and even on hot days we didn’t dare swim.

The treachery of this coastline is embedded into local legend.

Take the wreck of the Loch Ard further up the coast near Port Campbell in 1878, which killed 52 people. There were only two survivors: Thomas Pierce who was an apprentice midshipman and Eva Carmichael, a 19-year-old passenger. Wearing only her nightgown, Eva couldn’t swim and was found by Tom as she was drifting on some wreckage.

He rescued her, left her on the beach and climbed to the top of a cliff to get help. Tom found a station owner who went 10 miles home to get some rope and lifted Eva to safety. When they went to find her – she was no longer on the beach. She had found a cave and fallen asleep.

The pair spent the next few weeks recovering at the homestead, as news of their survival spread, and their legend grew. Eva was orphaned now and Tom was stranded. They should have fallen in love, right? They didn’t. When they recovered they went their separate ways but went on to live long lives.

• Brigid Delaney is a Guardian Australia writer and columnist