A Tory MP last week raised the delightful possibility that the big family of what we might call the terrorism community should be expanded yet further. Speaking to a group of activists at party conference, Jonathan Gullis declared: “The term ‘white privilege’ is an extremist term. It should be reported to Prevent, because it is an extremist ideology. It’s racist to actually suggest everyone who’s white somehow is riddled with privilege.”

Goodness. Even now I see it: online social studies graduates and right-on corporate HR functionaries hauling on the old orange PJs and trooping glumly into their cells in some British equivalent of Guantanamo Bay, alongside the murderous jihadists of IS, pipe-bomb prone neo-Nazis, incel spree-killers and armalite-toting provos. There may be something attractive in the idea, I’ll admit – report people to Prevent just for being a bit annoying – but I don’t think it’s really the way that elected MPs should be talking.

“White privilege” is no more than a jargonish phrase for something that we know to be true. The concept it denotes is that, all other things being equal, someone with white skin is on average going to find life a bit easier than they would find it had they had brown skin. It says: being white is not in itself an active disadvantage in life; and being brown – to a varying but still noticeable degree – can be. Far from being an extremist position, this is a point so uncontentious as to be a banality.

I italicised “all other things being equal” and “on average” because the defensive line on this – which seems to me a bit thick – tends to be to point to a white person or group of white people in a position of disadvantage and contrast them with a black person or group of black people at the top of the pole and declare: “Where’s their white privilege now, eh? Try telling this toothless laid-off manual worker that he’s ‘privileged’, eh? And look at that Kwasi Kwarteng – he’s a black chap and he’s a cabinet minister! You telling me he’s disadvantaged?”

Well, duh. All other things are not, in general or ever, equal. Class, sex, education, health, geography and sheer dumb luck all contribute to an individual’s situation in life. That’s exactly why demographers like to talk about population-level statistics rather than individual cases. On average, to take another example, men are going to be faster and stronger than women. The fact that Emma Raducanu could thrash me at tennis doesn’t alter this.

That’s also why the whole notion of privilege doesn’t stop with “white privilege”. The concept applies usefully to all sorts of characteristics. You hear of “male privilege”, “straight privilege”, and so on; and we don’t consider it contentious to notice that being born into wealth gives you a bit of a start in life. It’s just a way of designating any given characteristic you have that doesn’t – at population level and in and of itself – do you much harm. Why is it extremism to admit that such things exist?

To return to “white privilege”: those statistics on unemployment, pay differentials, health outcomes and the like are argued over. Many indicators are that they are improving. But the disparities are still there. Take the last report from the government’s own commission on racial equality – much criticised, incidentally, for painting a rosier picture than much of the left would have liked it to – published this spring. It found a 2.3 per cent pay gap between white people and those from ethnic minorities, found that 4 per cent of white people were unemployed besides 7 per cent of minorities, and so on.That may not seem huge, but it’s not nothing. And as far as the fine texture of life goes, it seems to be the experience of black and brown people that – less so now than fifty years ago, but still noticeably – they are from time to time made aware of their skin colour in a way that white people seldom are.

There’s no question that self-identified progressives sometimes use the term like a cudgel, as if “privilege” is a moral failing or disqualifies its holder from contributing to the conversation. That’s silly; and inasmuch as it puts their interlocutors angrily on the defensive, counterproductive. But the weaker case may be made: if you’re in the privileged gang in one respect or another you’d do well not to monopolise the conversation about that characteristic, because there are advantages you have that you will tend not to notice.

We can argue about why these things have come about and the best way of dealing with them. We can probably agree that things are in most respects travelling in the right direction. And we can drill down, usefully, into the ways in which “ethnic minority” and “white” cover huge and very varying subcategories and sub-populations for which outcomes can be radically different. But that’s the conversation – and even a backbench Tory MP would do well to recognise it rather than play silly culture-war games for cheap points from the faithful.

So white privilege exists, and it’s not racist to say so. You can ship me off to Gitmo now.