On Question Time last week, a member of the audience asked the panel about the racism scandal engulfing Yorkshire County Cricket Club. Fiona Bruce, in the chair, invited Nazir Afzal, the former prosecutor, to speak first. With indignation, Afzal replied, “the brown person will answer first.” When Bruce asked if it was wrong to have asked him before the other panellists, he said yes, “I think so”. Later Stella Creasy, a Labour MP, suggested that Bruce was “normalising forms of racism”.

In a single instance, we saw just how far through the looking glass we have travelled. Nobody seriously thinks Fiona Bruce is a racist. And nobody with an open mind believes she was “normalising racism”. If she had asked the white panellists to opine on racism before Afzal, she would have been accused of ignoring his “lived experience”, and allowing others to answer from the perspective of their “white privilege”.

Later, Afzal explained his position, saying, “the reality is that brown people, black people … are tired of having to continue to describe how much racism we experience on a daily basis.” Mirroring the now-famous book, much-promoted in schools, colleges and workplaces by equality consultants, Why I Am No Longer Talking to White People About Race, the suggestion seemed to be that it is wrong – perhaps racist, if you are Stella Creasy – for white people to even enquire of minorities about how to tackle racism.

Ironically, the issue debated on Question Time was the failure of white people to listen to an Asian whistleblower about racism. Azeem Rafiq, a former cricketer who gave evidence in Parliament last week, has revealed a culture of racism within Yorkshire cricket, and a refusal by the institution to listen to victims and to stamp out racist discrimination.

Complexity and controversy, and even hypocrisy and double standards, have since clouded the case. Shortly after Rafiq had given his testimony, stories about his own conduct began to emerge. First came the anti-Semitic language he used in exchanges with a friend on Facebook. Then came news he had posted a meme denigrating Africans on Instagram. Then came allegations that he had sexually harassed a sixteen-year-old girl.

In a way, the complexity ought to be helpful to us. The Rafiq case shows that victims of racism can also be perpetrators. And just because we know that Rafiq has been racist himself does not excuse the behaviour of those who were racist to him. Nor does it make the failures at Yorkshire any the less serious.

More important than the double standard shown by Rafiq, however, was the double standard of his supporters and other commentators in the media. When Rafiq apologised for his racist online posts, those who would normally be sceptical about such statements rushed to defend him. His behaviour was historical, they insisted, and he had shown remorse. Stella Creasy even praised “this poor young man”, for his “powerful, clear and compelling” apology.

Some sought to justify his behaviour, emphasising that his “teenage comments” could be explained by a lack of “social contact” and “dialogue” with Jewish people. When the young woman told her story of sexual harassment by Rafiq, the journalist who first reported the Yorkshire allegations saw only a conspiracy to punish Rafiq. “Weird that whistleblowers are so reluctant to come forward, isn’t it?” he asked.

What explains the double standard? It might, in part, be down to an understandable desire not to allow Rafiq’s own shortcomings to undermine his important revelations. It might, as some Jewish commentators observed, be explained by a hierarchy of racism, in which some care less about antisemitism than other acts of discrimination. But it is surely also down to blurred lines between different concepts: racial discrimination, racial inequality, institutional racism and the theory of structural racism.

Despite progress over the last few decades, racism undoubtedly remains a significant problem. Overt acts of racist discrimination – such as those exposed by Rafiq – and decisions made because of subconscious biases – such as in the abuse of police stop-and-search powers – must be tackled.

And inequality between racial groups might exist for reasons more complex than discrimination – economic geography, family structures or attitudes to women at work, for example – which must also be tackled.

The idea of institutional racism is controversial among many on the right, but the definition given by Lord Macpherson – “the collective failure of an organisation to provide an appropriate and professional service to people because of their colour, culture or ethnic origin”, seen in “processes, attitudes and behaviour which amount to discrimination through unwitting prejudice, ignorance, thoughtlessness and racist stereotyping” – rings true of what we now know about Yorkshire.

The theory of structural racism, however, is something else. It claims, without evidence, that society as a whole discriminates systematically against minorities and in favour of white people, causing minorities to suffer cumulative disadvantage. It is little more than a conspiracy theory inspired by French postmodernists and American sociologists, but its consequences are becoming clear.

For when we judge a specific act of racial discrimination, we can determine whether an individual is guilty or not. When we judge an organisation, we can decide whether it is living up to its responsibilities. But when we try to judge people according to a theory that already holds some complicit in systemic discrimination and others its innocent victim, all we get is a modern-day witch trial.

Nobody thinks Fiona Bruce is racist, but she is found guilty. We know Azeem Rafiq used racist language, but he must be innocent. Some find themselves forgiven for what others face excommunication. And the rules change, often retrospectively, to fit the theory, not the evidence.

Not only is this unfair, it is counter-productive. By pitting one group against another and declaring whole groups guilty, we are making it harder to tackle individual acts of racism, institutional racism, and entrenched racial disparities. We have to ask, how did we end up this stupid?