From open data to active data

A couple of weeks ago, I wrote about the problem of dormant data, and suggested that we now have the tools — policy, legal, cultural and technical — to transform dormant data into active data. In this post, I’m going to flesh out in a bit more detail exactly what this means.

Dormant data is data that’s open but radically underused. It’s hard to overstate how much trouble closed or dormant data can cause projects — from charitable projects, like my work at Cloud Break, to high-impact projects in fields like planning, architecture, engineering, construction, research and government policy.

These are the projects that shape our environment, economy and society, and they have historically been hampered by poor or difficult access to authoritative data.

Given the significance of these projects, it’s in all of our interests to make data access and reuse easier. This was why, a decade ago, governments around the world started to adopt open data mandates. By making data open, government agencies could quickly reduce the friction experienced by these high-impact projects. In addition, open government data could catalyse new innovative products and services, and lead to greater civic participation.

This is a transformative vision — and one that has yet to be fully implemented. In the first decade of the open data movement, many approaches have been tried, with varying levels of success. In general, though, too much government data has become dormant and underused.

The open data puzzle

So, we’re not yet realising the transformative potential of open data.

This is not necessarily anyone’s fault. As always, execution is key, and it’s only recently that all the pieces of the open data puzzle have come together. These pieces include:

• Cultural change, including evolving attitudes within government around the benefits of ‘open’.
  
• Government policy, including advocacy and education around the nuts and bolts of open policy implementation.
  
• Legal tools, such as human-readable open licences like Creative Commons.
  
• Technology, including cheaper cloud storage, better browser technology, and ultimately better ways to distribute data to those who need it.

We’re a technology company, and ultimately that’s where we put our focus. But it’s worth pausing here to note that the successful cycle of publication and reuse of open data — what’s becoming known as a connected data lifecycle — builds on a decade of heroic and painstaking work across government agencies, the private sector and civil society.

That work continues apace. But at this point in time, it’s becoming clear that a radical increase in data reuse is going to largely depend on improvements in technology. There are good reasons why the first generation of open data portals and catalogues look and work the way they do, and why so many users experience so much friction. But with smarter browsers, better bandwidth and many years of research and development, we’re starting see the emergence of something new.

A radical increase in data reuse

This ‘something new’ is predicated on the substantial reduction or removal of friction at every step of the publisher and user workflows. I’ve written about what this means in my post on the data lifecycle. When all the points of friction are removed — for both data publishers and all data users — we then see a radical increase in data reuse.

Instead of being open-but-dormant, we see the rise of what’s become known as active data. You can see some examples of this on the LINZ Data Service, such as the Property Titles layer (238,000+ views, 23,000+ downloads). Or, to take a less prominent example, the Waikato 0.5m Rural Aerial Photos (86,000+ views, 10,000+ downloads).

I think most would agree that this is a massive level of reuse for data in a relatively small country — and we haven’t even considered the usage of APIs and web services. 

Growing (and growing) the open data ecosystem

The reason why we at Koordinates focus on the ‘friction’ experienced by publishers and users is that not everyone is willing to endure that friction. Whether it’s a poor user interface, limited functionality, closed standards, high technical barriers, poor analytics or a time-consuming publishing workflow (or the other dozen things that make data publishing and reuse difficult) — every step in the process that isn’t easy or adding value will result in churn, i.e., users abandoning the process. This, in turn, means less data reuse, and less impact.

This needs to be underlined: We won’t realise the transformative potential of open data by publishing exclusively to and for the same old people. Let me take spatial data as an example: geospatial (GIS) professionals naturally see other geospatial professionals as their primary audience.

But the spatial data ecosystem is bigger than that. Much, much bigger. It includes researchers, architects, engineers, construction professionals, field workers, teachers, government officials, NGOs, house-buyers, conservationists, farmers, growers, planners…. I could go on (and on).

Let me say it again: Spatial data can travel far beyond the GIS industry. But not only that: spatial data must travel beyond the GIS industry if we are to realise the potential for both the spatial ecosystem AND open data.

The future of open data

This vision of active data is an ambitious one. And yes, it’s fair to say we’re not there yet. Even with the data services on our platform (like the LINZ Data Service), some friction remains. This is why our design team has been plugging away at the redesign of koordinates.com, which you can read about elsewhere on our blog.

But this issue is obviously much bigger than what we’re doing at Koordinates. It’s essential that everyone involved in open data keeps in mind the vision we shared a decade ago. We need to raise our expectations for what open data looks like and what level of impact open data ought to have.

It’s 2017. Simply ‘getting it out there’ is no longer good enough.

Image derived from NZ Property Titles from Land Information New Zealand, via the LINZ Data Service.  Original image made available under a Creative Commons Attribution 3.0 NZ licence.