With a new year comes the New Year’s Honours and I’m struck to see an MBE given to Pouria Hadjibagheri. He’s the technical lead of a civil service team whose drive and creativity led to the Coronavirus data hub. It was a breakthrough in the democratisation of public data. He and his team saw to it that information and metrics were not the secret preserve of a Whitehall cabal, cherry-picked to make a certain point, but available to everyone. This transformed the debate about the virus and the need for lockdown, allowing for new perspectives and new projects. The Spectator’s data hub was one of them. If you were pleased we avoided lockdown in December last year, you have Hadjibagheri’s team to thank – they made the debate possible.

The democratisation of data is a huge opportunity for an open society. Ministers have been promising this revolution for years but it never tends to happen: government departments are in no rush to have others mark their homework. ‘Don’t worry, minister, we’re working on it – looking good for Phase One in October 2024!’. Hadjibagheri and his team showed that world-class open data is affordable and doable in weeks rather than months, and to a standard of speed and accessibility higher than had been hitherto achieved anywhere in Europe or the US. It was a rare moment when the UK led the world, thanks to innovation from civil servants. It created a power flip. Before we knew it, Sage reports were no longer treated with just-down-from-Mount-Sinai reverence and instead challenged by all kinds of professional outfits.

But while the Covid dashboard was a sword-from-the-stone moment, the sword now lies discarded. Hadjibagheri and Clare Griffths, the team leader, could have been given the opportunity apply their skills to all government data, to throw the library door open. But instead their unit was disbanded and its members scattered. This is, to put it mildly, a missed opportunity.

Crisis breeds innovation, and the Coronavirus data dashboard was one of the most striking governmental achievements during lockdown. When the pandemic started, even Cabinet members were in the dark on basics like the latest estimate of the R-number. By the end, information was being shared not just with government but the public as soon as figures were available.  This allowed others to plug it into display devices, like our data hub. We were also able to join dots: show the percentage of Covid patients on ventilators, for example. Academics had instant access and even members of the public could check whatever they wanted, without needing a journalist to cherry-pick gloomy or Panglossian stats.

As transport minister, Grant Shapps was able to use the Covid dashboard to present his own case against lockdown at the all-important December 2021 Cabinet meeting. The work of Hadjibagheri, Clare Griffiths and others allowed both the man in the street and the man in the Cabinet equal access to the latest data. Britain had a richer, more balanced debate as a result.

For years, open data has been a basic government priority – but it has been a long time coming. When Transport for London made its bus and tube data available, allowing free apps like CitiMapper to come along, it transformed the information available to commuters. Government didn’t (or couldn’t) make CitiMapper, but government didn’t have to. All it had to do was create the API (so anyone could grab the data), release the figures, and let others do the rest. Nadhim Zahawi, a co-founder of YouGov and a big fan of data, summed this up in an interview with The Spectator when he was education secretary:

Transparency is your ally when you’re trying to reform or improve complex systems. We found this out when we pushed through [publishing] daily vaccine data. Publish it! Then whether it’s the Spectator data hub or a couple of kids in a garage, others can put an interface on it and analyse it in ways that actually challenges the system but also improves the system.

It’s often said that open data needs to be pushed across education, health and more. So what’s the holdup? What Hadjibagheri demonstrated was that it’s not just possible, but easy. Here are ten points behind his success, compiled with the help of my colleague Simon Cook:

1. Unlock the birds of data. They will fly on their own. Once data is accessible in the form of an API, it can be used on an app, a Bloomberg terminal, a Spectator data hub, universities, anything. It’s easy and free. As Zahawi was saying. So government’s first task is to create the APIs. By all means it can build its own dashboard, but anyone can make one of those. Only the government can digitise and release data in homogenous format.
2. Make the APIs easy for users to understand. The Covid dashboard team offered an API from early on but crucially added documentation and also built software development kits (SDKs) for major coding languages like Python, R and JavaScript within the first few months of the dashboard. These removed the necessity for people to build a lot of low level code converting the data into more usable forms (for example Python Pandas dataframes) and queries could instead just be a couple of lines of code. Over the life of the dashboard, they put together very good documentation on the various metrics. Clarity is all.
3. Make the codebase open source. The Coronavirus dashboard was bespoke but used Microsoft’s Cloud infrastructure (Azure), which simplifies deployment, scalability, resilience and infrastructure maintenance. There’s no need for government to reinvent the wheel.
4. Build a dashboard, showing your own KPIs, that can be used by everyone. Anyone who runs anything has key performance indicators (KPIs). Departments can have dashboards comparing their own progress vs results. There was actually is one of these in No. 10, set up in the Dominic Cummings era, but it is currently disbanded.
5. Power your data dashboard using your own data feeds. The Covid team did so, which helped a lot. To create the APIs is, in and of itself, the main revolutionary act: what you do with them is of lesser importance. So by all means make a dashboard, but use your own feeds. They are the main product. There is a humility to open data: knowing that others will join the dots. Your job is to supply the dots.
6. Start small (and simple) then build. The original Covid dashboard was simply case numbers and deaths. It later increased in scope hugely as more datasets were added. Don’t wait for months until it’s all ready.
7. Publish immediately. What made the Coronavirus dashboard absolutely vital was the immediacy of data. As soon as figures were available, they were published. So this did a revolutionary thing: abolished the notion of privileged access. No advance notice. As soon as the data was ready, it was available to the man in the street as well as the Sage committee.
8. Keep improving. The Covid dashboard team were active on social media – seeking and taking advice on improvements, flagging errors and explaining delays. They took up ideas such as adding an RSS feed and a ‘what’s new’ page. This built a lot of trust and a virtuous circle of improvement. Even the code for the dashboard is made public.
9. Take feedback seriously – and recognise the value of communication with the public. The Covid team took people’s feedback via social media seriously, discussing each and every suggestion and making changes or improvements accordingly. They also used Twitter as a platform for exchanging ideas with other experts.  One expert reached out on Twitter to say he’d written a SDK for .NET coding languages, based on the team’s design for Python: this code was checked and adopted, showing at-speed collaboration that is more Elon Musk than Whitehall. But Hadjibagheri showed that, if the right civil servants are empowered, they can be as entrepreneurial and open to new ideas as anyone in Silicon Valley.
10. Aim high – make data accessible to all, not just specialists. Crucially, they did not set out to create a Covid dashboard for ‘most people’. They aimed to create one for all people. User testing sessions were held and accessibility considered from the get go. Even colour blindness was thought about.

The Covid dashboard team worked with the speed and efficacy of Kate Bingham’s vaccine taskforce, doing things that had before been thought impossible. Previously, understanding local public health hit a roadblock in that local government administrative areas (so-called MSOAs) were not aligned with NHS trust and NHS region areas. The Covid team didn’t just unify this data, but went further and mapped them to postcodes. Then they published it as an API. So now, thanks to them, the codes for every postcode is now on public record (here’s the House of Commons, for example). So how did they manage to do this when years of public health data project managers had not? The Covid data team didn’t have more money. They just had the right remit and an emergency mandate: a licence to move fast (across departments) and innovate.

In a private company, a lab team who invent something amazing are asked to scale it up – only in Whitehall would they have given a gong and dissolved. After 15 years of banging on about open data, here is the Tesla of open data – developed not by Silicon Valley but a team of civil servants. So why not roll it out across Whitehall?

The Sunak government should recall Hadjibagheri and have him create a far bigger data hub, covering everything. Think what a difference could be made by the ability to process and publish massive amounts of data at very short notice with the highest standards, high accessibility and in a consistent framework. Analysts, journalists, decision makers, etc would all know where to go – and how to get quality data feeds. This would allow the biggest problems confronting government to be approached from more angles, and more solutions offered. It would cost almost nothing. What’s stopping them all?

The government could create dashboards for other crises

Civil contingencies planning could be updated so a Covid-style dashboard would be set up whenever we had a flood or another crisis, like in energy. Dashboards could monitor experiments such as ULEZ in London. We do our best with The Spectator’s data hub but we have modest resources. Right now we’re trying to make up for the lack of APIs by using Python to create automatically-updating series, bringing together various sources. It works, but it’s clunky. Government could automate all this in a stroke.

An open data revolution is within Sunak’s grasp. The PM has, in Hadjibagheri and his colleagues, a dream team that can be reconvened any time with a proven ability to work at pace. He badly needs good ideas. The more open and accessible government data becomes, the more ideas he’s likely to get.

Pouria Hadjibagheri’s MBE is well deserved, but it should be the start of something, not the end. His work has taken the UK government to the threshold of a new era in unleashing data for the public good. Reconvening that team and letting them take their project to the next stage would be a good prime ministerial New Year’s resolution for 2023.