James Joyce’s celebrated biographer seemed a mild man to fellow academics – but his ambition and steely self-belief made him a callous husband and father