‘Festival?’ said Nathan Milstein. ‘What is festival?’ I had naively asked the most immaculate of violinists where he used to play in the summer and he looked at me as if I had proposed an unnatural act.

‘Before the war,’ said Nathan, offering a glimpse of paradise lost, ‘Volodya and I would stay at Senar for six weeks with Rachmaninov.’ Volodya was Horowitz, his best friend.

‘In those days,’ he continued, ‘we liked to spend time with composers. A composer was someone you could talk to. He knew philosophy, literature, lepidoptery. Rachmaninov could name all the butterflies around Lake Lucerne. He liked me better than Volodya, maybe because I was not pianist.’

Milstein was a one-off, a cavalier who played 30 recitals a year at most. His contempt for festivals and routine concerts was uniquely refreshing. Now, ahead of a music-free summer, it feels acutely relevant. Do we really need festivals?

The model of the modern music festival was shaped 100 years ago this summer by Richard Strauss, Hugo von Hofmannsthal and Max Reinhardt with the notion that artists could play for fun and each other at Salzburg, far off the beaten career track. In the post-war Austrian rump, the idea seemed at once noble, useful and lucrative.

The only summer competition was Bayreuth, which was never so much a festival as an orgy of Wagner worship, woefully reduced by the 1920s to penury and fringe politics. While Salzburg built a concert hall, Wagner’s son Siegfried and daughter-in-law Winifred embraced Adolf Hitler. Fat Ruhr cats soon followed.

A man who owned much of East Sussex built a summer festival in 1934 for his small-voiced soprano wife, Audrey Mildmay. Sternly professionalised by the Hitler refugees Fritz Busch and Rudolf Bing, Glyndebourne achieved high standards of Mozart performance and a certain shabby chic among the moneyed classes. Its owner, John Christie, aspired to outshine Salzburg, but never did.

The only other pre-war festival was Lucerne, which came into being when Salzburg flew the swastika. Georg Solti would say he owed Lucerne his life when Toscanini took him on as an assistant in 1939, saving him from his family’s Holocaust fate. The Proms were never spoken of in those days as a summer festival; they were just a two-month run of concerts with chaps in spats who sauntered around the Queen’s Hall smoking Sobranies and flitting off to dinner before the symphony started.

The festival industry, as we know it today, began in 1947 when Rudolf Bing envisaged ‘a platform for the flowering of the human spirit’ in Edinburgh, a city that made Sunday a leaden misery and viewed theatre as the devil’s work. The Edinburgh Festival took off with a £10,000 gift from Lord Rosebery, whose horse won the Jockey Gold Cup. While Bing imported Bruno Walter and the Vienna Philharmonic, students ringed the official festival with fringe acts that mocked the frock coats and laid the foundations for 1960s satire. Edinburgh was the first festival where young people set the tone. Aldeburgh, where Benjamin Britten pitched his rival festival tent, was always old before its time, soured by minor musicians who competed for the composer’s bleak favour.

Back in Europe, Bayreuth was strutted by Emmy Goering as a Nazi widows fest while Salzburg came under the thumb of its native son Herbert von Karajan, a conductor who aimed to control the music world. Karajan was titular head simultaneously of the Berlin Philharmonic Orchestra, the Vienna State Opera, the Philharmonia in London and half of La Scala, also dominating the profitable record industry. Between the mid-1950s and 1980s, the Salzburg Festival — along with two extra events that Karajan added at Easter and Whitsun — was his Roman Colosseum where fresh talent was thrown to the lions and record bosses came bringing gifts. At the Karajan festivals I attended the air stank of sycophancy, cheap Sekt and fear; even taxis were festooned with his corporate advertising.

Lucerne, meanwhile, waxed fat on Nazi gold and the highest ticket prices on the classical planet. It built an exquisite concert hall that seemed to float on the lake and took pride in contracting the socialist Claudio Abbado to entertain its preponderance of plutocrats, many of them on the wrong side of 80. Rachmaninov’s former villa glowered bleakly across the lake.

Up in the mountains, a Swedish agent called Martin Engstroem started a festival that, he swore to me, would avoid corruption and ostentation by forming a community of artists in the original Salzburg ideal. For a summer or two, his scheme flourished in a marquee that shook in a breeze and obliterated the finer points of orchestra music. His festival was located at Verbier, winter playground of the idle rich, and it was not long before vanity trumped verity. Today, with the Putin apologist Valery Gergiev as music director, Verbier is an oligarch magnet and Engstroem has set up offshoots in Latvia and Georgia.

Briefly, in the 1980s, music looked to Schleswig-Holstein for renewal. The pianist Justus Frantz put on concerts in barns and farmyards in a bovine province without a concert hall to its name between medieval Lübeck and the outskirts of Hamburg. Hearing an atonal string quartet by Peter Eotvos while perched on a bale of hay, I fell in love with music all over again. But the costs were too high.

In Oxfordshire, an English banker weighed down by Saudi petrodollars turned Garsington Manor into Glyndebourne-lite. Countryhouse opera was 21st-century chic. A producer named Wasfi Kani set up opera festivals in Hampshire (Grange Park Opera) and Leicestershire (Nevill Holt) and built a 700-seat new theatre in the North Downs. Longborough in the Cotswolds put on a Wagner Ring — Bayreuth without the bad vibes. While Covent Garden and English National Opera failed to nurture British talent, these festivals incubated a new generation.

More music festivals flourished each year, to the point where there were 300 in France alone. I saw the most chilling Peter Grimes of my life in sleeting rain in the Finnish castle at Savonlinna. I found a Mahler festival in the Dolomites, a Shostakovich shrine in an east German resort, a devotional music festival in Morocco, a Janacek festival in Brno and chamber music in the Jerusalem YMCA, curated by Elena Bashkirova, Barenboim’s wife. At the Arena di Verona, beside Lake Garda, big voices belted out their last pay day before booking into the Casa Verdi for retirement.

And suddenly they’re gone. Every last one of them with the exception of a shrunken Salzburg opera and a possible week of Proms in an empty Albert Hall: gone as if they never were, and amazingly unmissed. I shall yearn for the flight of evening swallows at Garsington and the three choirs at Worcester Cathedral as cricketers flicker on the field outside. But there’s no cricket either this season, and that I shall miss even more.

The summers of music have gone swanky, stale and sullen. There are too many festivals offering too little by way of difference and innovation. The BBC Proms are festive only on the last night. Festivals lack fun, not to mention surprise. If there is to be any bonus from this Covid blight, let it be the space to reconceive how we spend our future musical summers.