A number of peers made the call during a debate on Monday on the Counsellors of State Bill, which is being fast-tracked through Westminster.
The upper house heard a proposal to add Anne, the Princess Royal, and Prince Edward, Earl of Wessex, to the list of "counsellors of state", who can stand in for Charles III when he is overseas, on an official trip or ill.
There are currently five counsellors - Camilla, the Queen Consort, along with the four most senior adults in the line of succession: the Prince of Wales, the Duke of Sussex, the Duke of York and his daughter Princess Beatrice.
Watch: Prince Andrew and Prince Harry royal roles challenged in parliament
However, Labour's Lord Berkeley called for Harry and Andrew to be removed from the list of stand-ins as they are non-working royals.
Andrew has withdrawn from royal duties because of his association with US sex offender Jeffrey Epstein and underage sexual assault allegations, while Harry has stepped back from being a working royal and lives with his family in California.
Lord Berkeley said: "The most important thing for me is the question of whether the Duke of Sussex and the Duke of York will continue.
"The Duke of Sussex is abroad, as we all know. It is pretty clear that he is away for quite a long time and I question whether he should still be on the list.
"The Duke of York no longer undertakes royal duties, I understand, so I assume that he is excluded from being a Counsellor of State.
"The bill quite rightly adds two more members so, presumably, it can also exclude two members who, I suggest, are no longer working members."
Conservative peer Lord Balfe said: "What the bill is doing, I think, is trying to deal with an act that was conceived and passed before the idea of a working member of the Royal Family was invented.
"That makes for part of the difficulty because it clearly does not remove either the Duke of Sussex or the Duke of York from the list."
Liberal Democrat peer Lord Newby added: "My Lords, the British constitution is an extremely strange animal. The bill shines a light into one of its darkest corners.
"How many of the general public know that there are such things as Counsellors of State? How many could name them?
"A number of noble Lords have suggested that we ought to have a root-and-branch look at who might be Counsellors of State.
"One can think of ways in which the situation could be easily improved — for example, inserting the word “working”, albeit with some appropriate definition, to cover those members of the Royal Family who would be eligible to be Counsellors of State."
The bill was debated after the King requested adding the Princess Royal and the Earl of Wessex as he marked his 74th birthday, to “ensure continued efficiency of public business when I am unavailable”.
The bill will add Anne and Edward to the list, but stops short of removing Andrew and Harry.
However, the House of Lords heard that only “working members” of the Royal Family would be called upon to act as Counsellors of State.
The bill will complete its remaining stages in the House of Lords on Wednesday before heading to the Commons, but the government is not expected to accept the proposal to remove Harry and Andrew from the list.
Watch: Princess Royal and Earl of Wessex to deputise for the King