Realising the vision of open data

When the open government data movement was born in the mid 2000s, it had a vision of open data driving a new era of global prosperity. This was a pretty grand vision, but it was founded on the basic fact that government data is crucial to countless high-value social, economic and environmental projects—the projects that literally shape our planet.

But underlying this vision was the assumption that by ‘getting data out there’, governments would see a radical increase in data reuse. On that front, the open data movement still has some work to do. But before we talk about the future, let’s quickly go back to the beginning.

Like anyone else who has been working in the geospatial industry over the last decade, I’ve had my share of difficulties accessing data for projects. When I started working in the industry, most of the data I needed was closed-by-default.

This usually meant that agencies had to manually extract data for their customers from their internal systems, copy it over to DVDs and hard-drives, and then send it through the post (courier pigeons having recently been decommissioned).

It wasn’t the most efficient system. This process was a major hassle for both government agencies and data users — which meant that high-value government data was hugely underused.

A partial solution to this problem arose in the mid-to-late 2000s, when governments around the world began to seriously commit to releasing their data openly. Given this starting point, the initial focus of the open data movement was to ‘get it out there’ — ‘there’ being on the internet, under an open licence. The primary goal was to remove the barriers to open data, and make everyone’s life a bit easier.

In the years since, the open data movement has evolved considerably. We’re seeing many more government agencies move beyond the old ‘get it out there’ approach. Taking advantage of new technologies, agencies are raising the bar and working to publish data better, with the aim of generating far greater levels of high-value reuse.

But what does ‘better’ open data look like?

Start with the user

It’s no secret that a well-designed process or product should start with the user—how they work, how they think, where they carry out their tasks, etc. By designing around the user, it’s possible to dramatically reduce the barriers faced by new users of your process or product.

Over the last thirteen years working for government and industry, I’ve been around a huge number of people hunting for data to use. So in the spirit of good design, let’s start with the common patterns I've seen there. The workflow of most data users can be broken down into four general steps.

Find

Users tend to search across a wide range of data sources, validating as they go that the data is from an authoritative source, such as a government agency or reputable vendor.

Appraise

Users need to know what they’re getting before they’re able to use the data with any confidence. In the spatial world this usually involves previewing the dataset by itself (including interrogating the data to ensure it has the required features and content), and layering it with other relevant datasets to see how the combined output looks.

Access

Geospatial datasets can be truly massive and are often complex, so users tend to ‘crop’ multiple data layers to their area of interest. And because the application of spatial is very broad, the file formats required are also broad — from established GIS formats like SHP, to non-GIS formats like DWG and geospatial PDF.

Use

More technical users will often want to connect with data via APIs and, for geospatial professionals, OGC-compliant WFS and WMTS web services. This streamlines access, and enables the creation of new products and services.

What about the publisher?

To date, open data products and processes haven’t usually considered the way data users actually work, which has placed an unnecessary burden on users. But the same is true for data publishers. While data publishing workflows are highly variable, depending on the size and nature of each organisation, we can identify a range of common pain-points and bottlenecks.

Import

Agencies need to be able to add data directly from their internal data sources, and automate regular updates via API. They also need data to be cleaned and verified on import, to ensure users don’t experience errors.

Store

At the platform-level, agencies need their data automatically SEO-optimised, and stored in such a way that it can be previewed and repackaged on-the-fly for data users.

Publish

Agencies need their own branded site, with built-in licensing, metadata and granular access controls for private data sharing of non-open data.

Insights

Before publishing more open data, agencies need rich analytics to understand user behaviour and make informed investment decisions.

A connected data lifecycle

From what we’ve seen there is now an opportunity to radically reduce the ‘friction’ of both publishing and accessing open data. By connecting the workflows of data publishers and data users together, a connected data lifecycle is formed.