BAck in September, Boris Johnson had a meeting with some of the 21 Tory MPs who were about to lose the party whip over their opposition to a no-deal Brexit. He was warned that he was prepared to risk the hardest of Brexits, he might lose crucial parliamentary seats – reliably Tory places where a majority backed remain. One of the people present was Anne Milton, the MP for Guildford, the largest in the world in 2017 with 55% of the vote and a 17,000 majority. She says she has expressed her fears of defeat, only to have them thrown back at her. "If Guildford's lost, it's lost," said the prime minister, and that was that. Two months later, Milton is standing as an independent, Guildford may well be a four-way marginal – and one of the most fascinating stories is rolling on.

In the face of all the noise about "Workington Man" and the purported significance of leave-voting in the north of England, this particular subplot has been overlooked. It is about the extinguishing of David Cameron's Tory "modernization" project – and the growing disconnect between voters in affluent Conservative heartlands and the ludicrous ideology of Jacob Rees-Mogg and the European Research Group.

While the Tories have been transforming themselves into the nationalist, Pooterish party of hard Brexit, essentially cosmopolitan attitudes – open, internationalist – have been seen in the formerly conservative areas of our cities, and the suburbs and towns around them.

Some of this was evident in 2017, in the seats lost by the Tories to both Labor and the Liberal Democrats: Bath, Kingston and Surbiton, Battersea, Enfield Southgate. The same story is reflected in the weakening Conservative presence around Manchester and Bristol – and Lib Dem targets such as St. Albans in Hertfordshire. But perhaps the most interesting story lies in Surrey, which means Tory England.

Until recently, the county had 11 Tory MPs. But by the time the election was called, the Conservatives had expelled Anne Milton, Philip Hammond and Sam Gyimah (now a Lib Dem). In local elections this May, the Tories lost 117 local councilors in Surrey and control of local authorities. Among Surrey's eight remaining Tory MPs are such Brexiters as Dominic Raab, Michael Gove and Kwasi Kwarteng. But in the borough of Guildford, remain got 56.2% in the 2016 referendum; in Woking, 56.2%; in Elmbridge, which includes Raab's seat of Esher and Walton, nearly 60%.

'If Guildford's lost, it's lost,' Anne Milton said the prime minister told her.

'If Guildford's lost, it's lost,' Anne Milton said the prime minister told her. Photograph: Steve Meddle / Rex / Shutterstock

I went to Guildford last week. Of late, it may have seemed rock-solid Tory. But way back in 2001, the Lib Dems took it with a majority of 538 – which, combined with an uptick in 2017 and recent winnings in local elections, seems to have convinced them they can win again. Two years ago, Labor got close to a not-unreasonable 20% of the vote. Obviously, the Conservatives are still in danger of being reckoned with. But now that Milton has been thrown out of the Conservatives and is running a campaign based on "a smartphone and a laptop", there is a clear sense of what has happened: it's the election in microcosm.

When I talk to people in the center of town and watch some people in the upscale neighborhood of Onslow, there's a lot of people talking about it. 2016 Conservatism had left them feeling abandoned. "Labor and the Tories are too extreme," said one man, who has been loyal Conservative. "I want a centrist party." These days, "centrist" is usually thrown around as an insult; he said it with a sense of pained loss. A little earlier, I had a churchgoing couple in their 60s – he has a geologist, she has self-described housewife – who had so voted Conservative down the years, but were also switching. "It feels like we're voting on Brexit, and we're stayingers," they said.

It is easy to still think of a suburban mindset – prim, reactionary, moralistic and deeply conservative, with both small and large Cs. The massively popular 1970s sitcom Terry and June, a camped-up glimpse of a couple trying and failing to live the perfect middle-class life, was set in Purley, on the London / Surrey Borders. In the songs of those ageless sounds of Surrey, the Jam, there are portraits of characters who fit the same picture: Mr Clean ("Getting pissed at the annual office do / A smart blue suit, and you went to Cambridge, too" ), and Smithers-Jones, the hapless commuter who "stops off at the corner of the shop to buy the Times", turns up to work as usual and suddenly gets the sack.

Immerse yourself in these songs, and you hear a world that endures, but is also in the midst of change. The population of Surrey is becoming more ethnically diverse, and is coming from London, they are bringing a more cosmopolitan culture to them. On Guildford's north side, the University of Surrey draws in a multitude of people from overseas, and lots of students from London. The town's support for a blunt economic question: if your livelihood is dependent on the capital's booming economy, why would you vote to leave the EU?

And so to another question: what does post-Cameron, Rees-Moggist, Brexit-or-bust Conservatism really have to say in the world? Perhaps the usual Tory warnings about Labor plans will be cut through; Jeremy Corbyn will do its work. But through the seemingly endless Tory mishaps of the last couple of weeks, I started to sense that the ground is under way. When there is an accident-prone, there are usually deep reasons why. The sense of conservatism weakening in its own heartland is certainly some of the explanation.

This story may not be enough in the future. What is happening can be more slowly and unpredictably than that. But any Conservative with their eye on the Tories' long-term prospects would be well advised to think very hard. Parts of England are drifting away from a political tradition, which is a matter of instinct, but it seems to be lost in a murk of nostalgia and Brexit dogma. And whatever the daily hurly-burly of the election campaign, it is these things that tend to decide whether or not political parties have a future.

John Harris is a Guardian columnist