From a temple to Athena, it became a Byzantine, then Latin, church, a mosque, a powder magazine and finally a ruin. Lord Elgin’s vandalism was hardly anything new