Former prime minister Lord Salisbury referred to Balmoral Castle as "Siberia".
Another, Benjamin Disraeli, complained that "carrying on the government of the country 600 miles from the metropolis doubles the labour".
Tony Blair described his annual visit as "a vivid combination of the intriguing, the surreal and the utterly freaky" - and said he only survived the weekend with the help of strong alcoholic drinks.
His youngest child, Leo, was conceived at the home after Cherie Blair left her contraception in London due to the "sheer embarrassment" of the staff unpacking her wash bag the previous year.
And David Cameron, in contrast, recalls the "bliss" of leaving his close protection team behind as he walked the hills with his wife Samantha.
"The Queen is keen that you should have complete solitude," he wrote.
But Liz Truss will not meet the monarch for a relaxed summer break, as her predecessors have done.
Instead, the new Conservative leader will fly to Aberdeenshire to be formally appointed prime minister - an event which usually takes place at Buckingham Palace.
This is referred to as the "kissing of hands", and in recent times involves only a bow or curtsy and a handshake.
The audience is expected to last half an hour, although Gordon Brown's appointment in 2007 was followed by a "congenial and business-like" discussion lasting 58 minutes.
Clement Attlee's meeting with King George VI in 1945 was rather more brief due to the shyness of both men. After an awkward silence, Mr Attlee opened the conversation saying: "I won the election."
"I know," the monarch replied. "I heard it on the six o'clock news."
Prime ministers' visits to the private castle in the Scottish Highlands date back to Queen Victoria's reign and typically take place over a weekend in early September.
Balmoral estate was bought by Prince Albert for his wife in 1852, and features monuments to Victoria's consort, children and her close friend and servant John Brown.
The current Queen invites guests to use her highland ponies to explore the hills and glens. Grouse shooting, deer stalking and salmon fishing are also on offer.
The climax of a prime ministerial holiday is the "Bothy Barbeque", previously presided over by the Duke of Edinburgh and held in a stone hut originally built for shepherds.
"The Queen drives you at breakneck speed across the moor to a bothy," David Cameron wrote in his autobiography.
"The Duke of Edinburgh is outside, tongs in hand, smoke rising from a row of sizzling grouse. And then the two of them cook and serve you dinner.
"Literally, the Queen of the United Kingdom and Commonwealth Realms topping up your drinks, clearing up your plates and washing up."
Historian Ben Pimlott described Margaret Thatcher's presence at such a gathering as: "Monarch and consort cooking sausages for the disconcerted premier and her husband on a windswept hillside - each couple trying desperately to be informal."
Ms Truss will likely need to survive in office until next September if she is to receive such an invitation.
Her visit to Balmoral is, however, an opportunity to begin a relationship that some of her predecessors have found instructive and invaluable.
Ben Pimlott describes the Queen's role as that of "constitutionally sanctioned counsellor and therapist… the only person a prime minister talks to whose confidence he knows will not be abused".