Charles Duff’s memoir tells a sad tale of cruelty and betrayal with spry wit rather than bitter resentment. Notwithstanding the subtitle’s threat of earnest Welsh soul-searching, Charley’s Woods is tart, arch and crisp. It recalls a strange, lonely childhood with brisk frivolity and a ruthless perception of other people’s oddities, vices and humours.

Duff was born in Battersea in 1949. His mother, Irene Gray, was a Dublin social worker who pioneered role-play therapy in Ireland, and became pregnant by an Irish don of French-Jewish descent. After her son’s surreptitious birth, she hastened back to Dublin. In a hugger-mugger fashion, without legal formalities, two collateral royalties, the Marchioness of Carisbrooke and her sister-in-law the Countess of Athlone, arranged for the infant to be adopted, at the age of ten weeks, by Sir Michael and Lady Caroline Duff.

The Duffs had a mariage blanc. He preferred men, and her taste was predominantly for women. She had, however, become pregnant, and Duff, a country neighbour who wanted a male heir but was unlikely to father one, married her for mutual advantage. Some attributed the paternity of Caroline’s child to her uncle-by-marriage Duff Cooper; but Anthony Eden, who had been her occasional lover since the 1930s, believed he was the father. In any event, the baby was stillborn. Jonathan Gray was substituted as a consolation for Caroline, and transmuted into Charles Duff. His new forename was in memory of his adoptive mother’s recently dead father, the Marquess of Anglesey.

Michael Duff was the last male of a family of Welsh slate millionaires originally called Duff-Assheton-Smith. They lived in a large, plain, white-faced house, Vaynol, in Gwynedd, with a 1,000-acre park enclosed by a seven-mile wall. The park had a manmade lake with three islands, red deer, white cattle, a rhinoceros and a giraffe. Mount Snowdon was part of the property.

As a character sketch of a disturbed baronet and of father-son animosity, Charley’s Woods vies with Osbert Sitwell’s account of his impossible father. Sir Michael stuttered, seemed witless, had an elephant fetish, hated horses and adulated royalty. ‘He looked so odd: tall, thin, shapeless, with a gormless smile on his handsome face, and, sometimes, foam around his mouth,’ Charles recalls. His callous spite towards a child whose adoption he regretted shows an astounding lack of self-awareness.

Caroline Duff lost her virginity to Tallulah Bankhead, who is quoted as saying: ‘Pity the poor lesbian, who cannot whistle at her work.’ She lived for decades with a tipsy actress whom Charles remembers as ‘wild, rebellious, insecure and wonderful fun’. Dressing in slacks and men’s sweaters, Caroline was ‘opaque, mysterious and intriguing’, Charles says. ‘There was an incomparable stillness about her. She always listened beautifully — although I’m not sure she often heard much. Her movements were lithe and sexy.’ She was a splendid mother to a small boy — ‘affectionate, tactile, encouraging; often tickling, laughing and rolling about on furniture and floors’. But she rejected her foundling after he reached adolescence. Her wayward self-indulgence is ultimately chilling.

Charley’s Woods is rueful rather than boastful. It abounds in lordly and theatrical anecdotes, waspishness and mordant intelligence. After reading the works of Dietrich Bonhoeffer and Teilhard de Chardin, the teenage boy was buggered in his changing cubicle on a Tangiers beach by Joe Orton (‘It was the first time I had had sex with anyone other than a schoolfriend’). There is grateful remembrance of obscure noblemen, glamorous cosmopolitans such as Cecil Beaton and of the Vaynol chauffeur and his wife, George and Eluned Russell, who saved Charles’s sanity as a child (‘I loved them as much as I loved anybody’). He is loyal in his admirations: he was at prep school with Prince Charles, and now thinks him ‘the reincarnation of Solomon’.

There is a delicious section on a man who doubled as salesroom correspondent of the Observer and as a pimp for upper-class married gay men. Originally named Harry Carnes, he wished to change his name to ‘Simon Sailor’, but being dissuaded, compromised by becoming ‘Simon Fleet’. Duff says that Fleet woke each morning thinking ‘what has this day got to offer me, and for me to offer others?’ It might be the epigraph for his tender-hearted, prickly, resilient and life-enhancing memoir.