Conservative lawmakers — and even some of his Cabinet — are worried that Prime Minister Rishi Sunak’s plans for stabilizing the U.K. public finances will sink their chances of surviving the next election.

One Cabinet minister said he’s concerned that people will conclude Sunak is accepting Britain’s decline with his determination to raise taxes and cut spending as the economy enters recession. That’s the sort of idea that can define a prime minister, they said. And with public services already under pressure, it may well mean the Labour opposition extends its 20-point lead in polls.

Sunak and Chancellor of the Exchequer Jeremy Hunt are looking to find savings of about £50 billion ($56 billion) when they set out their budget plans on Nov. 17, an official familiar with their thinking said last month. That would be enough to plug a £35 billion gap in the public finances and give them some room to maneuver as they prepare for an election due by January 2025.