‘Tourism, tourism and tourism,’ said my Antiguan cab driver, when I asked what the country’s main industries were. Still, it’s easy to avoid the other tourists, even though the island’s just over 100 square miles. Take a quad-bike tour — arranged by my hotel, the Sandals Grande Antigua Resort — and you can go from one end of the island to another in a morning, without seeing another tourist.

Instead, you’ll see fields of sweet potatoes, dotted with sprawling tamarisk trees; jagged cliffs and pale-yellow beaches, fringed with luminous, aquamarine water. You’ll also come across remnants of old sugar plantations; in the early colonial years, slavery was Antigua’s biggest moneymaker.

Most stirring of all is Betty’s Hope Estate, Antigua’s first major sugar plantation, founded by Christopher Codrington in the late 17th century. I felt a little chill when I read, in the pretty little museum there, that Codrington’s son’s legacy paid for Hawksmoor’s library at All Souls, Oxford. I once spent many happy hours in that library, reading in the shadow of Christopher Codrington’s statue. I didn’t realise my pleasure was subsidised with slave money.

Divine retribution, in the form of a hurricane, has swept away the Betty’s Hope mansion. The slave village has been swallowed up by undergrowth. But the windmills that ground the sticky cane still stand, their robust machinery imported from Derby.

A happier legacy of colonial rule is Antigua’s capital, St John’s, its yellow and pink pastel houses clustered beneath the baroque towers of St John’s Cathedral. Just outside St John’s, the old sugarcane fields are dominated by a worthy replacement: the Sir Vivian Richards Stadium, named after Antigua’s greatest cricketer and largely built, like the new airport, with Chinese money.

Loveliest of all is Nelson’s Dockyard in English Harbour, on Antigua’s southern tip. This hurricane-proof inlet was home to Nelson from 1784 to 1787, when he was enforcing the Navigation Acts, preventing trade with the new United States of America.

The dockyard — a classical complex of copper and timber stores, sail lofts and shipyards — is enchanting; a chunk of Georgian England teleported to the Caribbean. The little Dockyard Museum is good on Nelson — who found his wife, Fanny Nisbet, on the nearby island of Nevis. The Admiral’s Inn, in a splendid old house, does a fine risotto.

You’ll bump into more of the island’s main modern import — tourists — at Shirley Heights Lookout, the hilltop bar in a former military base above English Harbour. It’s popular on Sunday nights, when a steel band plays Bob Marley songs as the sun goes down over Montserrat. Antigua lies on the cusp of two seas: look west and you see the calm, warm Caribbean; look east for the cool, choppy waters of the Atlantic — and Eric Clapton’s villa and recording studio, strung across the spine of a seaside peak.

Just a short walk away is the cemetery to the men of the 54th Regiment (2nd Battalion Dorsets) who died in the West Indies between 1840 and 1851. It was a pretty grim posting, with the humid heat, yellow fever and intense self-medication with local rum. Today, the cemetery is haunted by dozens of tiny hummingbirds.