Rishi Sunak goes into the summer holidays in the same position he began the year: 20 points behind in the polls. In other ways it feels as if his premiership has gone backwards. Mortgage rates have risen above the levels they were under Liz Truss. The Tory psychodrama of the Boris Johnson era has led to two of the three by-elections taking place this week. Little progress has been made on Sunak’s ‘five priorities’ – the junior doctor strikes show no sign of abating and the Rwanda scheme is held up in the courts. ‘At this point Keir Starmer could probably announce backing for freedom of movement and still scrape in,’ says one downbeat minister.

The by-elections this week – in Uxbridge, Selby and Somerton and Frome – are adding to a sense of impending doom. ‘One was sparked by a Tory MP taking cocaine so we probably deserve it,’ says a government adviser anticipating defeat. Across the country, Tory MPs are adopting the brace position, either saying they won’t stand again or (as in the case of Ben Wallace) announcing in advance their intention to resign from cabinet.

Yet the political landscape is beginning to change as next year’s general election gets closer. Scrutiny of Labour’s plans for government is growing – and will continue to do so. This week Starmer faced his biggest clash yet with his backbenchers over his insistence that a Labour government would not scrap the two-child benefit cap.

The decision is based on a price tag – £1.3 billion – which Starmer’s shadow chancellor believes she cannot commit to. Rachel Reeves, the most influential member of the shadow cabinet, takes the view that she cannot afford to make more spending commitments for fears that the Tories will attack Labour as reckless on the economy. Just whether this is the case (the scheme would amount to about a thousandth of the £1 trillion state spending) is open to debate. Many Labour MPs think the decision is more about priorities than about cash.

At shadow cabinet this week, Starmer referred to Truss’s unfunded tax cuts as evidence for why he could not commit to abolishing the benefits limit. Starmer has also said spending needs to be paid for by economic growth, not by more debt or significant tax rises. Originally, he claimed growth would come via £28 billion a year of green spending. That plan has been delayed due to high borrowing costs, so his main hope will be planning reform. Labour MPs may be waiting a very long time indeed for that to deliver significant growth.

So the Labour MPs frustrated at what they regard as Tory parsimony may not see much change under a Starmer government. Reeves has done her best to temper expectations. She used a speech last year to warn that a Labour government won’t be able to do many of the things it wants to in its first term. Only this week have the implications of this dawned on many MPs.

Unlike with Tony Blair and Gordon Brown, who inherited low debt, low tax and a roaring economy, Starmer looks set to inherit a sluggish, debt-addled economy taxed more than any in peacetime history. Aides talk of the need to keep pressing the idea that this is all down to Tory failure. ‘The Tory legacy needs to be our version of the “there’s no money left” note attack they put on us,’ says one figure.

Starmer had previously called for the two-child cap to go, so this U-turn has been embarrassing for him and his shadow cabinet. ‘They are starting to understand what a difficult decision is,’ says one minister. While resignations have been avoided, MPs across the Labour party have lashed out. There have been fiery scenes outside the weekly meeting of the parliamentary party. One Labour MP joked that the anger and mayhem was like a return to the Jeremy Corbyn era. These scenes have raised questions as to whether Labour could really hold the line in government.

Starmer has even won a new nickname on social media: ‘Sir Kid Starver’. ‘It’s only going to get more difficult,’ warns one party figure. With up to 18 months until the next election, Starmer and Reeves will have to make more difficult decisions while also trying to keep the party united. No. 10 hopes this is the start of a wider Labour crack-up. One shadow cabinet minister takes the view that there is little risk in moving the party to the right, as voters are so keen to oust the Tories they are unlikely to defect to the Green party. It’s a theory that is about to be heavily tested.

Sunak’s plan is to keep calm and stick to his five priorities, rather than admit defeat and try again. ‘A reset rarely works,’ says one government aide. During the Cameron-Osborne era, reducing the deficit was the first step to achieving anything else. Sunak says that curbing inflation (which, unlike the deficit, he cannot directly control) is what will create room for the Tories next year. Inflation fell by more than expected last month and is predicted to keep dropping this year, albeit perhaps not quite fast enough for Sunak to be confident of hitting his target of seeing it halve.

Sunak will outline a fresh final-year mission at the Tory party conference in October. He will have a new cabinet, made up of MPs hungry to win the next election. He will use the King’s Speech to set out further both what his government hopes to do before the next election and what it would do if the Tories somehow won a fifth term. His agenda is likely to include leasehold reform, as well as a focus on fraud and energy security (which the Tories believe is a clear dividing line with Labour).

Starmer has good reason to feel more positive than Sunak this summer. His various U-turns and disputes over policy have not hurt his party’s poll rating: almost half of the country say they intend to vote Labour. But for him, holding the line will only get harder. If he wins, the toughest chapter will then begin.