When we judge politicians’ views, why should their skin colour be in any way relevant?

There were so many ways to criticise Kwasi Kwarteng last week. The authenticity of his ethnicity was not one of them. The MP Rupa Huq lost the Labour whip after calling Kwarteng “superficially black” at a Labour conference fringe meeting – an unacceptably prejudiced conclusion to draw from her observation that he has more in common with other privately educated colleagues than with her black constituents on a housing estate.

Because I was chairing that fringe meeting, co-hosted by British Future and the new Black Equity Organisation, I felt compelled to challenge those remarks, asserting that Kwarteng was black British, black African and recognisably black. The audience applauded this warning that Labour needs a clear red line between politics and prejudice, reinforced by the race activist Chantelle Lunt. A simple golden rule would help. Don’t criticise a black Conservative in ways you would not criticise a white Conservative. If a critique of Kwarteng could be made of, say, Jacob Rees-Mogg too, carry on. If it depends on his race, desist.

Huq told me she was “mortified” by her use of “superficially”. Some on the left do defend ugly racial slurs like “coconuts”. I do not believe she is among them. Some of her problems arose from how carelessly she spoke about race. The garbled half-sentences in a stream of consciousness about a “very multi-culti Tory leadership contest” stumbled into casual stereotypes. One of the worst directly echoed David Starkey’s comments about David Lammy a decade ago: that he does not sound black on the radio.

It is a damaging narrow stereotype to cast educational and professional success as white attributes if black people do not fit one particular stereotype. The Labour leader, Keir Starmer, immediately suspended Huq. This seemed one example of “cancel culture” that almost everyone could agree on. What happens next? I have long thought that the point is not to cancel but to change. We should work for a public culture that welcomes meaningful apologies and tries to make some use of them. Political parties rarely do that, instead letting people slip quietly back before some crucial vote, making vague references to diversity training that sounds like box-ticking.

The best antidote to casual prejudices is meaningful contact. Civic and party groups could help to facilitate this. Those like the Black Equity Organisation and particularly black-led Conservative groups could help the MP to understand how her comments were received. Albie Amankona, of Conservatives Against Racism For Equality, says black Conservatives would welcome the chance to talk – and debate ideas, not characteristics. He thinks common ground could be found across parties, from teaching our history better to narrowing ethnic disparities, if we got beyond debating who really counts as authentically black or brown.

It is important that ethnic diversity becomes a “new normal” across parties. That is what integration looks like. Black and Asian voters have less power if one party thinks their votes are in the bag and another that they are unreachable. About a quarter of ethnic minority Britons vote Conservative, and about six out of 10 for Labour. But any sense of entitlement from Labour is toxic. Recent political history is the stories of a “core vote” – from Scotland to the “red wall” – that got fed up with being taken for granted. The first generation of Commonwealth migrants identified strongly with Labour but that has faded away. Most young voters start with little impression of what either major party stands for.

David Cameron, who began with just one black and one Asian Tory MP in 2005, sought to catch up with Labour’s stronger record on representation. Now Conservative progress puts Labour under pressure over diversity in leadership. Despite having twice as many ethnic minority MPs as the Tories, it is less likely to see them in top roles. Why? Our fringe heard too many examples of black councillors feeling patronised by white Labour colleagues, especially in discouraging them from standing outside ethnic minority-majority wards. Being celebrated as a community tribune can become a pigeon hole.

That ethnic minority Britons were only half as likely to vote Conservative as white British people was a key motive for Cameron’s diversity drive. But few people vote on candidate characteristics. Older voters certainly did not vote for the 70-year-old Jeremy Corbyn. Theresa May did better with men and worse with women than Cameron had done. The cabinet’s makeup is unlikely to be the key driver of ethnic minority votes either. Representation matters in a multi-ethnic Britain – but substance matters more.

Ethnic minority Conservatives can be caricatured by allies as well as opponents. The most stridently anti-woke voices always get plenty of airtime, but are only a small fringe of the Tory minority vote. British Future’s research shows that many more Black and Asian Conservatives are centrist balancers who want constructive action on race equality accelerated. Half of ethnic minority Tories supported the Black Lives Matter protests, while a fifth were critical. Most think “white privilege” remains a real thing in British society today – though a quarter do not – yet also think calling Britain “systemically racist” is too simplistic.

In aggregate, ethnic minority Tory voters have views quite close to those of white Labour voters. Both groups are part of a latent consensus for more action on race equality – from tackling CV discrimination to challenging online hate – that spans political divides about how to talk about race.

Making ethnic diversity a new normal helps us assess people by results. When we judge a chancellor in office, it’s the economy, stupid.

• Sunder Katwala is director of British Future and former general secretary of the Fabian Society

• Do you have an opinion on the issues raised in this article? If you would like to submit a letter of up to 250 words to be considered for publication, email it to us at observer.letters@observer.co.uk