‘Why this sudden restlessness, this confusion?’ asked C.P. Cavafy in his poem ‘Waiting for the Barbarians’:

Because night has fallen and the barbarians have not come.
And some who have just returned from the border say
there are no barbarians any longer.
And now, what’s going to happen to us without barbarians?
They were, those people, a kind of solution.

All through your and my life the Labour party have been at the gates of Downing Street, and often enough stormed them, only to be beaten back at a subsequent election. What might happen to the Conservative party if those barbarians disappear?

We must not assume that Jeremy Corbyn will take the Labour leadership. The likelihood remains that when second preferences are counted Yvette Cooper or Andy Burnham will scrape through. But theirs would be a miserable victory: humiliated before they even begin. Their party now faces one of two alternatives: a real victory for Mr Corbyn, or a Pyrrhic victory for Ms Cooper or Mr Burnham. The voting begins next Friday, 14 August. Burnham says the Labour party may split and I do not doubt him.

Tories will at first rejoice. The barbarians are fighting among themselves and no longer threaten us! Hurrah! And it is true that a Labour civil war or even disintegration would guarantee the Conservatives’ return to office in 2020. Shall I, then, live (d.v.) under a Tory prime minister until I’m at least 75?

There are reasons to doubt it. At our peril do we underestimate the way Labour’s shadow lurking in the wings has bound Conservatives together. No internal gravitational pull by any one uniting ideal keeps our always-troubled Tory marriage alive: it’s fear of what lies outside our walls that achieves this.

When, aged 20, I arrived at Cambridge in 1969, I did not join the university Conservative association. CUCA was all cravats and ghastly sherry parties, jockeying for office and gossip about big Tory names in London. I knew only that I was not a socialist, that the trade unions were ruining Britain, that I believed in the individual, and that Labour and its levelling and collectivist instincts were part of our country’s problems, not their solution. So I joined two political groupings: Pest (Pressure for Economic and Social Toryism) and the Cambridge Liberal party.

At Liberal meetings I observed that the party was not serious about power. Thus — and purely by elimination of the alternatives — I ended up as a Conservative.

It’s an unspeakable old beast, the Tory party: I often disagree with it and sometimes come close to despair. But I string along, in fear of the barbarians. How many of my country-men — MPs, party members as well as Tory voters — have followed the same path to our party’s gate? Tens of millions, I’ll bet. On the doorstep at this year’s general election my fellow activists and I did sense a surge of support for our candidates, but which of us can honestly say we were smothered by love and trust for our party? We cannot. We cannot claim even that people were much interested in our policies, or knew what they were. Overwhelmingly the doorstep response of our ‘Tory definites’ was ‘Look at the polls! We’ve got to keep those buggers out!’

Those buggers, as Cavafy might put it, have been a kind of solution. Remove fear from the Tory equation, and what holds us together? I’m far from claiming nothing does, but the Conservative party really has three cores. Each to some degree repels the others. Each, shorn of the others, could attract support from outside the party. I go along with some of my Times colleague Tim Montgomerie’s analysis of the political groupings that would better match 21st-century Britain, but my own formulation follows.

There’s undoubtedly a Ukippy strand in Toryism — and there’s more of them in than out. They are, in good and bad ways, somewhat reactionary. They’re deeply patriotic, fierce about defence and hostile to the EU. They’re morally conservative (no gay weddings for them). Their instinct is to support the bosses rather than the workers, they hate tax and are not overly sensitive to the woes of the poor, but they’re not wholly free-marketeers. An orderly market rather than a free-for-all appeals to them. For this group I shall use Tim’s name: the National party.

Then you have a gang that Michael Gove’s former adviser Dominic Cummings could lead, small in number but strong in the power of their energising philosophy. They’re dominated by dislike of government and bureaucracy. They have confidence in the unshackled market. They read Hayek. They were patronised (but less often heeded) by Margaret Thatcher. I’ll call them the Small State party.

Finally you have what I’ll call the One Nation party. We’re a bit wishy-washy, economically fairly liberal but quick enough to intervene if things go wrong; and we worry about the poor. Morally we’re on the permissive side. We may not be mad keen on the EU, but on balance we’d let well enough alone.

These three groups are philosophically further from each other than (say) Ukip and the Tory right; or the ‘Orange Book’ Cleggite Lib Dems and the Tory left. But all co-exist within Conservatism in often-uneasy alliance. Were Labour’s barbarians to depart the gates, then events — those reliable creators of gangs — would soon weld and trademark them. The One Nationers would draw supporters from former Lib Dems and the shattered Labour party. The Nats would bring the kippers back into their fold. The Small Staters would probably migrate both ways: influential more as a diaspora than a movement of their own.

I would see no need for any other political parties in Britain, except for the real left: the Corbynites. And we should be back to three parties again, but parties that better match the nation we now are. The kaleidoscope would have been shaken. We may well see the result.