His first impression of Oxford University in 1958 was of ‘effortless wealth and privilege everywhere’ and, feeling like a foreigner, he pined for the familiarity of close-knit Wigton