The future of open data publishing

Data, data, everywhere

Over the last decade, we’ve witnessed a transformation in the distribution of public data. Government agencies around the world, from urban and regional authorities to state and federal governments, are now releasing their data on the public internet, for anyone to access and use. Generally speaking, this data is made available free of charge and most copyright restrictions on use.

The reasons for doing so are clear. A paper published by McKinsey in 2013 estimated that open data could generate more than $3 trillion a year in additional value for the global economy. Similar predictions have been made by governments around the world.

And the good news is that there has indeed seen a sharp increase in the amount of authoritative data released by government for free and open reuse. While it hasn’t always been smooth sailing, open data publishing is gradually becoming business as usual for many countries around the world.

The transformative potential of open government data

So, what’s the problem? Well, the thing is—and this might sound counter-intuitive—the goal of the open data movement was never just ‘release more data.’ The actual goal was to see a sharp increase in high-value use of authoritative data, across civil society, industry and government itself.

And getting data used is hard. While the open data project has been successful in getting more open data on the internet, it has struggled in generating the levels of use required to realise its initial vision (I’ll talk about why this is the case later on). Every open government data project in the world has found it difficult to bridge the gap between ‘getting the data out there’ and ‘getting it used.’ While there are some great open data case studies, these are still relatively isolated.

It’s fair to say that we haven’t yet realised the full, transformative potential of open government data.

The particular case of geospatial

This is particularly the case with geospatial data, which has the greatest potential to transform our society and economy, but has also the greatest barriers to high-value use.

Geospatial professionals know these barriers all too well, particularly the low-level data wrangling tasks, such as sourcing and translating spatial datasets published across a range of government websites and servers. These rudimentary, somewhat painful tasks have, unfortunately, become part of the job.

But what about those who lack geospatial training and software? What about the multitude of architects, engineers, analysts, designers, draftsmen, researchers and more—all those involved in the projects that determine how we understand and shape our planet?

For these folks, the process of accessing and using data can be truly daunting. Indeed, lacking specialist software and training, simply finding, downloading and appraising large and complex geospatial datasets from a variety of sources and translating them into the formats they need—such as DWG, or geospatial PDF—is nearly impossible.

For the open government data project to realise its potential, we need to find a way to reduce these barriers. Otherwise, industry, government and civil society will continue to bear the heavy transaction costs of accessing and using authoritative geospatial data.

The workflows of data users

To tackle this problem, we need to go back to basics, and examine the real-world workflows people are using to find and access data. This sounds like a complex task, given the sheer diversity of potential users. But for our purposes, we can simplify the geospatial data workflow into four general steps, as our Head of Design Dan Newman outlined in his design research blog: Find → Appraise → Access → Use.

The aim, for government agencies publishing open data, is to reduce the bottlenecks and pain-points at each of these steps. These pain-points include:

1. Find: Data is scattered across a multitude of servers, catalogues and portals.
   
2. Appraise: Data often needs to be downloaded in its entirety and imported into a specialist software application before users can tell if it’s useful for their project.
   
3. Access: Data is made available in specialist formats that require training or expensive applications to translate and combine.
   
4. Use: Data lacks OGC web services and APIs to enable the creation of new products and services.

As you can imagine, these are difficult problems to solve: geospatial data is sourced from various local and national government agencies; it can be large in size and complex; and the industry uses a range of open and proprietary file formats that can be hard to translate and combine.

To make this even more difficult, whatever solution public agencies adopt needs to be designed for both geospatial and non-geospatial professionals. This solution is something we’ve called authoritative data publishing—that is, publishing on a platform that has been designed and engineered to meet the technical requirements and workflows of both data publishers and data users. 

More on that in our next post.