Ever since Boris Johnson’s disastrous decision to try to stay the standards committee’s guilty verdict against Owen Paterson, things have started going wrong for Downing Street. The roots of the government’s problem can be traced back to a speech, though. Not Johnson’s rambling address to the CBI earlier this week, but his speech at Tory party conference last month.

Johnson wrote much of the speech on various plane flights. David Cameron’s conference speeches, in contrast, were normally the result of long, painful writing sessions for him and his aides. Yet one advantage of writing the Prime Minister’s speeches by committee was that it forced everyone involved to think through what the government’s priorities were, what it was actually trying to achieve.

Johnson’s conference speech was the ultimate Boris Johnson address: good jokes, but not much policy. It went down a storm in the hall and few politicians could have pulled it off. But it didn’t actually clarify the government’s priorities.

It is not a coincidence that Downing Street’s bad run has overlapped with the resumption of normal politics after Covid. ‘The public are very forgiving of the fact we have been through a pandemic. But we are in danger of losing that goodwill,’ frets one long-serving aide. The last few days of headlines won’t have helped. Yet the real worry raised by Johnson’s CBI speech isn’t that he lost his place or made peculiar references to Peppa Pig but that he was trying to give a joke-laden speech to an audience of stern businessfolk and that this was only one of three significant speeches he was giving that day. It suggests the Prime Minister’s time is not being well spent.

Ministers were struck by a recent presentation by the Tory strategist Isaac Levido to the cabinet a fortnight ago, in which he stressed that the next election would be much more like 2015 than 2019. Levido’s argument was that the three key factors which led to the Conservative victory two years ago were Brexit, Jeremy Corbyn and Boris Johnson — but the next election campaign would be a far more traditional affair. If the Tories want to win, they will have to place a far greater emphasis on delivery and what local MPs have done to help their constituents.

Corbyn is gone. And while Keir Starmer’s personal polling numbers might not be impressive, they are, as Levido pointed out, still a significant improvement on Corbyn’s. Corbyn also served another purpose for the Tories: he united their new electoral coalition. In crude terms, red-wall voters disliked his lack of patriotism and his cultural leftism, while more traditional Tory voters in the south feared the effects of Corbynomics.

In 2024, Brexit will not be the issue it was in 2019. The polling Levido shared with the cabinet showed that in marginal seats in the last election, it was a key issue for more than six in ten voters; that number has dropped to just a quarter of the electorate. Johnson has talked about the importance of a ‘keep Brexit done’ narrative for the next election, yet it is hard to imagine that by then Brexit will have anything like the resonance it had in 2019 after months dominated by attempts to overturn the referendum result.

The declining salience of Brexit is a particular problem for the government given the numbers of illegal migrants crossing the Channel in small boats: already three times more have arrived this year than last. This is causing alarm among ministers. One Johnson ally laments that small boat crossings are making ‘a total mockery of our attempts to say we have taken back control.’

In a sign of how the Channel crossings crisis is exposing the Tories’ right flank, Nigel Farage is openly speculating about a political comeback based on this issue. If he were to return to the political scene, he would immediately knock several points off the Tories’ poll ratings — sending Tory MPs into a panic.

Tax is causing problems for the Tories on the right too. The increase of National Insurance to put more money into health won’t solve all the problems with social care, hence this week’s Tory revolt on the question of how the cap on care costs is calculated. These problems could well grow. On Tuesday, Johnson had to downgrade his manifesto pledge from the idea that no one needing care would have to sell their home to pay for it to the idea that no one will be ‘forced to sell a home they or their spouse is living in as it will not be counted as an asset’. Raising taxes with a promise to ‘fix’ social care is one thing, raising taxes to make incremental improvements is quite another.

These issues are hurting Johnson’s standing with the party activists. One senior Tory MP reports that when he was the guest speaker at a constituency dinner this month, one party member opined that ‘this is not a conservative government’. The MP was taken aback by how many of those present agreed with the sentiment. In part, this is a risk of governing from the centre — the party’s own voter base can feel neglected. But it is worth remembering that after a year and a bit in office, Tony Blair had a list of achievements to reassure the left of his party: the minimum wage, union recognition in workplaces where a majority of workers wanted it, and a 25 per cent increase in aid spending. Johnson has one big achievement to please his base — Brexit — but he needs other things to add to this list. As this senior Tory MP complains, hiring more police officers and nurses isn’t a ‘fundamentally differentiating’ issue between the Tories and Labour.

It isn’t just Tory party members who are grumbling, MPs are too. There are obviously plenty of backbenchers who have never liked Johnson and feel that recent events have given them licence to cause trouble. But discontent has spread beyond this group. One veteran MP warns that ‘colleagues are starting to feel that there’s no theme to this government’.

It doesn’t help that the relationship between No. 10 and the whips’ office has been damaged by the Paterson fiasco. No. 10 would be well advised to move quickly to repair this breach. When there’s obviously division between the two, parliamentary discipline begins to rapidly unravel.