William, then 15, and Harry, 12, lost out on time with both Diana and father Prince Charles while the pair were going through their divorce.
Harry told the Diana, Our Mother: Her Life and Legacy documentary that a combination of the divorce, his mother’s charity work and her romance with Dodi Fayed meant that they had not seen each other for weeks.
The Princess of Wales died in a car crash in August 1997 – the day before they were set to be reunited.
Harry said: “The two of us were bouncing between the two of them and we never saw our mother enough or we never saw our father enough.
“There was a lot of travelling and a lot of fights on the backseat with my brother, which I would win.
“I don’t pretend we’re the only people to have to deal with that. But it was an interesting way of growing up.”
He added to an anti-landmine campaigner: “You saw my mother more recently than I did.”
Diana had spent ten days with her sons and Mr Fayed at his villa in the south of France before the princes returned to the UK.
Their mother went on to join her boyfriend on his yacht in the Mediterranean before travelling to Bosnia to campaign against landmines.
She remained in phone contact with her sons and Harry recalled his final conversation with his mother before she was killed.
He said: “It was her speaking from Paris, I can’t really necessarily remember what I said but all I do remember is probably regretting for the rest of my life how short the phone call was.
“Looking back on it now it’s incredibly hard, I’ll have to sort of deal with that for the rest of my life.
“Not knowing that was the last time I was going to speak to my mum, how differently that conversation would have panned out if I’d had even the slightest inkling her life was going to be taken that night.”
Diana, Our Mother: Her Life and Legacy will be screened on ITV on Monday at 9pm.
Top pic: Rex