Two weeks ago, like most people, I hadn’t so much as heard of ChatGPT. By last week, I was hearing of practically nothing but. After OpenAI released its large-language model chatbot for the public to play with, it passed a million users in five days flat. Hype poured in. Columnists asked it to write the opening paragraphs of their columns about ChatGPT – with, of course, hilarious results.

Educationalists worried that this new AI chatbot would render coursework redundant and require a return to timed exams, so swift and plausible are its responses to prompts like ‘write me an essay about the causes of the first world war’. Computer programmers reported that it was ace at writing passable but not strictly functional bits of computer code. The Nobel Prize-winning economist Paul Krugman asked simply: ‘Will robots take away our jobs?’

By last night, so overwhelmed was it with trivial requests from drunk people, it had hoisted the flag of surrender, like Twitter’s famous ‘fail whale’. Of course, that error page was noisomely cutesy: ‘Explain the status of ChatGPT as a sea otter. Squeak squeak! Sorry, ChatGPT is very popular right now. Please try again later! Squeak squeak!’

It struggles especially with poetry – which, sure, so do most humans

Still, I was hyped. I was one of those people bombarding it with trivial requests. And in some respects, it really is quite something. I cannot speak to the information architecture that makes it possible, but it’s obviously top notch. You can admire its speed and fluency – and I dare say it would ace the Turing test from time to time, which is no small achievement. It produces wholly grammatical, natural language responses and has an impressive ability to work with what appear to be abstract concepts.

But it’s still just… very so-so. It may be impressive compared to, well, most of what has come before. It may, indeed, supply the odd shortcut for people looking to gussy up a first draft of a piece of technical writing to work from. But its responses – composted probabilistically from a giant corpus of language – are not human-like unless you ignore all the actual things that humans do. Those are the difficult things such as tone, diction, cadence, irony – and understanding concepts rather than chucking them together in plausible conjunction. (It might be useful in that respect to academic literary theorists but most end-users, including even faintly clued-in secondary school teachers, would smell a rat.)

It also has very weird gaps in its knowledge. It produced a decently clear explanation of quantum superposition, no question. It was able to make broad-brush remarks contrasting the styles of William Empson with Cyril Connolly, or Stephen King with James Herbert – and got both those things spot on. So sometimes, as you’d expect, it draws on information it has gleaned from the web. But at others – especially if asked a direct question – it primly denies that it has any capacity to search the net for information at all.

I asked it to write a Petrarchan sonnet about hamburgers and, though it evidently likes hamburgers, it clearly has no idea what a Petrarchan sonnet is; it produced a sort-of Shakespearean sonnet with the rhymes all wrong and unscanned lines. I asked it for poems in the style of W. H. Auden and Walt Whitman and e. e. cummings and it gave me poems by Auden and Whitman and cummings. Make it write a poem about cheese in the style of Emily Dickinson – surely not the hardest person to spoof – and it produces some doggerel about cheese without the faintest hint of Dickinson’s diction.

I asked it to write an essay on race in the work of Ursula Le Guin, and it produced an excellently plausible – if rather vague – argument about the importance of race in her work, drawing on evidence from two or three of her books. But a central plank of its case was the whiteness of the protagonist of A Wizard of Earthsea, and how he grew as a character by learning to rest on his moral qualities rather than his racial privilege. If you hadn’t read A Wizard of Earthsea, you might have turned that essay in to your form teacher. If you had read A Wizard of Earthsea you’d know that its protagonist Ged has red-brown skin. ‘Didn’t you get that wrong?’ I asked it. ‘Oops,’ it said, or words to that effect. When I asked it to write the same essay 24 hours later it avoided mentioning that book altogether. It’s basically, I concluded, a world-class bullshit machine.

But it does not have, not one bit, what David Foster Wallace writing about tennis called ‘touch’. Oddly, given that language itself is its big thing, ChatGPT simply has no feel for stylistics whatever. Its default prose is pure Wikipedia. It has redundancies and clumsy repetitions. It struggles especially with poetry – which, sure, so do most humans. And when I asked it to write a piece of erotica in the style of Alan Bennett (don’t kink-shame me; I know more than half the population dream furtively of the same thing) it was… not impressive. I can’t see Bennett ever writing the words, ‘she was wearing a tight-fitting dress that hugged her curves in all the right places, and I felt a sudden surge of desire coursing through my veins’.

What OpenAI has produced seems to me to be a marvellously clever toy, rather than a quantum leap, or a leap of any sort, in the direction of Artificial General Intelligence. It’s a very sophisticated Magic 8 Ball. Tech hype is always like this. Remember how, for about twenty years, Richard Branson has been about to send passengers on commercial spaceflights with Virgin Galactic? And how, like disappointed millenarian cults revising their schedule for the apocalypse, the date has always been a year or two in the future? How Elon Musk’s scheduled landing on Mars has been pushed back from 2024 to 2029?

What we’re seeing here, I think, is not a level assessment of ChatGPT’s achievements, any more than our excitable reporting of the claims of a Branson or a Musk reflect the probability that we’ll all be orbiting the earth in a Virgin spaceship or planting crops on Mars any time soon. The hype reflects the excitement: the desperate wish for something to come down the pipe which will be magical, transformative, world-changing, astounding. And if that thing doesn’t show up, we’ll pin our hopes on something that at first glance looks like it might be. The wish is father to the thought. Which is, it should be said, the sort of uniquely human folly that no AI could reproduce.

The post ChatGPT: a world-class BS machine appeared first on The Spectator.