For three decades I have been dreading the day Malcolm Fraser would die. Not because I was a huge fan; quite the opposite. Having written an infamous article in the Australian soon after the Memphis missing trousers story – ‘my night with Malcolm’ – I knew that I would have to revisit the unpleasant incident once again. And so it passed that soon after Malcolm died, the calls came from writers wanting the full sordid story.

Having criticized the commentators who attacked Gough Whitlam in the hours after his demise, I had declared that in the first 24 hours after a PM’s death we should keep it nice for the sake of the family. As luck would have it, I was filling in that very day as co-host of Sky’s afternoon show with Ross Cameron so I thought I could safely avoid lauding Fraser and leave that to others while remaining circumspect. Instead I arrived to discover Cameron’s comments critical of his former leader had gone viral and some colleagues suggested perhaps I might want to provide some balance by being a bit sympathetic to Mr Fraser.

It was a big ask but as the day wore on I learned I need not have worried about bagging the man, as there was no shortage of detractors; the most virulent coming from his own side of politics. Well, his original side of politics – the right.

And while many in the left were lauding him, some maintained the rage.

Few would be surprised to hear former Labor minister Graham Richardson phoned in to declare Mr Fraser ‘the worst post war PM in Australian history’.

And I always thought Billy McMahon had a lock on that.

When I asked independent Senator David Leyonhjelm about Fraser’s death he replied bluntly ‘I have to say I’m not sad. My memories of him are mostly bad’.

His complaint was the one most oft heard from conservatives; that Fraser’s government was one of ‘missed opportunities’ because he came in with massive expectations in the landslides after the defeat of Whitlam.

And the criticism was not just coming from the more extreme right, those that became incensed that Fraser grew into a raging leftie in his later years.

Business has been down on him ever since he left office and well before he turned hard left. Their mantra has always been that he ‘wasted his mandate’. He was a PM who controlled both houses of parliament and never did the big reforms they were seeking.

Sure the papers have trotted out a long list of achievements from founding the Federal Court and SBS to allowing in thousands of Vietnamese refugees. But that is not what you call real economic reform in their book. They seem to forget that at the time of the 1983 election not all corporate chiefs were heralding the change. Some well known business figures were threatening to leave the country if former ACTU boss Bob Hawke became PM. Renee Rivkin was one who followed through on his threat but returned from London when he realized that not only was Hawke not another Whitlam but he was thankfully not another Fraser.

Ah, with the hindsight of history the great Hawke/Keating economic reforms are a shining example of the missed opportunities of the Fraser years. Again let’s not mention that the basis of the revolution was the Campbell Committee report into financial deregulation, which was a product of the Fraser Government.

The fact Fraser did not implement them immediately is obviously the cardinal sin and a message that messrs Abbott and Hockey would be best to remember as their own financial reports pile up – Shepherd, Harper, Murray….

It could also be argued that the chaos and lingering divisions in the wake of the Dismissal probably made Fraser more timid as he tried to settle a skittish nation.

Perhaps there is another warning for the current Abbott government given you would think it was not hard for Fraser to look better economically than the previous Whitlam government which was reduced to taking paper bags of illicit money from Arab loan sharks. For someone like me who started in journalism in 1979 when Fraser was still the Great Satan to ALP members, it was truly bizarre to watch a teary Bob Hawke extol his virtues. (Or is Bob racking up credits with the other side, being the main character referee for Liberal Treasurer Joe Hockey at his recent defamation case?)

When we came to the ‘winners and losers’ segment of the show I noted the big loser to me was Malcolm Fraser. Unlike his contemporary conservative leaders in the ‘80s, Thatcher and Reagan, who in death were elevated to lofty heights, Fraser was receiving a comparably lukewarm reception on the day of his death. Goodness knows what they will say about him in a week’s time, I asked.

Well, they did a good job trying on the next Monday in a day of condolences in parliament. His side showed respect more than affection, even from Mr Abbott whom Fraser detested so much he ripped up his Liberal party membership in 2009.

Of course the next Malcolm for PM, Mr Turnbull, was the most eloquent and effusive probably noting the parallels with a small ‘l’ Liberal leader, one with a social conscience who did not pander to the right. But given how many Coalition members privately complained they didn’t even want to be there listening to the parliamentary tributes, it is clear Fraser’s legacy will always be complex.

And as for that story about a young 24 year old reporter with the Oz and her encounter with the former PM in 1986 when he had only been out of office a mere three years and was still a towering figure? I went for a drink with a man who declared he wanted a ‘dark smoky smoochy bar’; a man who began by asking what drink got me drunk. Hey, it was Malcolm Fraser and obviously not your usual lothario. He was head of the UN Eminent Persons group on apartheid at the time. But the night ended with a New York cabbie watching me fend off this large man in the back seat and declaring ‘lady you look like you are in trouble’ and pulling up to let me escape at the first corner.

That was how the night ended and would have been merely a bizarre dinner party anecdote until Fraser went and lost his pants in Memphis and claimed he was drugged or rolled in an upmarket establishment. I thought I could illuminate him and others on what might have happened. I think her name was Peaches.