The year 2019 seems a long way away. Whether or not David Cameron can stay in office until then is this week’s hot topic of conversation among Tories. They wonder how many more weeks like the last two the Prime Minister can endure. Before Parliament broke up for Easter, the view among Cameron loyalists was that the Tory party needed a holiday. The thinking went that the recess would remove MPs from the Westminster pressure cooker and let referendum tempers cool. But this break turned out to be a disaster. The government spent the first week trying to get on top of the Port Talbot steel story and the second attempting to fend off the fallout from the Panama papers.

For Cameron himself, the break was a reminder of how hard it is for any contemporary leader to have anything approaching a normal life. First, he had fellow guests snapping and tweeting his movements on his break in Lanzarote. Then he had the media in full hue and cry about his late father’s business dealings. The publication of his own tax returns seem to have put a stop to this story. But at some cost. A major breach has been made in the principle of taxpayer confidentiality. What has started with the Prime Minister, the Chancellor and the leader of the opposition will not end there. We will all repent these actions at leisure. (Do we really want to live in a society where everyone knows what everyone else earns?)

Meanwhile, Cameron widened Tory divisions on Europe by approving the spending of £9 million of taxpayers’ money on a government referendum leaflet plus website and social media strategy to support staying in.

One senior backbencher who knows the mood of the party as well as anyone tells me that the PM’s ‘departure will be much sooner than is generally acknowledged’. Indeed, some of those in his circle are already bracing themselves for an attempted leadership challenge after the EU referendum on 23 June, whatever the result.

However, any attempt by his critics to remove Cameron straight after a vote to stay in, however narrow, would be a mistake; it would look like sour grapes. One bullish cabinet minister tells me that if the rebels try anything in the summer ‘they’ll lose by a mile’. But, revealingly, experienced backbenchers increasingly view some kind of challenge to Cameron as close to inevitable. ‘There are an awful lot of cross people who don’t like him and know he doesn’t like them,’ one cautions.

Yet the events of the past few weeks make it more likely that Cameron will try to carry on for some time. He has long been determined to be the first Tory chief since Stanley Baldwin in 1937 to leave the leadership of his own volition. He even asked a historian friend to find out for him which Toryl leaders have quit at a time of their choosing and not one determined by the electorate, their party or their health. If he were to leave soon after the referendum because of divisions in the party, he would not achieve this aim.

If a leadership contest took place ‘so shortly after the referendum there’d be quite a big element of rejecting what’s gone before’, one of his confidants admits. To put it another way: in a contest held in the shadow of the EU campaign, an Outer would probably triumph. This would change the Tory party, and not in the way Cameron intended.

One of the things he is keenest to do is to ensure that the party he leaves behind is one carved in his own political image. He was delighted after last year’s Tory conference because he felt that both Boris Johnson’s and George Osborne’s speeches had endorsed his brand of one-nation Conservatism. Six months on, the picture looks very different. Boris has come out for Brexit, an act which has tainted Cameron’s view of him. And George Osborne is faced with the difficult task of rebuilding his political standing for a second time after a torrid few months.

As those closest to Cameron admit, he needs to get a sense of what he stands for more firmly embedded in the public consciousness. So, with the Queen’s speech and after the referendum, there will be a return to the social-reform agenda that he set out in his conference speech. This will take time. One confidant says: ‘He doesn’t believe that if he goes in this year or next he’ll have achieved what he wants to do as PM.’

To stay in No. 10, though, Cameron must reunite the Tory party after the referendum. At the moment, he is not making that easy for himself. He is campaigning with no thought for the feelings of those in the party who disagree with him. It is one thing for a leader to disagree with close to half of his MPs and most of his activists, but quite another not to do it respectfully.

There is little sign, however, that Cameron intends to change his approach to the referendum. ‘He’s very worried that without a very energetic, general-election-style leadership from him it might well be lost. That would be a far bigger disaster for him than any party issue,’ remarks one pro-EU minister.

Cameron has always taken the approach that he’ll deal with crises as they arise, and he clearly plans to do that with the fallout from the referendum. But reuniting the Conservative party will take more than a ‘reconciliation reshuffle’. It will require the articulation of a clear Tory agenda that his MPs can unite behind.