Introducing Scans and Geospatial Data Sources

Koordinates supports a range of data sources, including ArcGIS Rest APIs, PostgreSQL, Amazon S3, & WFS.


Posted by Hamish Campbell
August 14th, 2019


This is part of an ongoing series covering less well-known features of the Koordinates platform. 

Well over 40,000 users access geospatial data across the Koordinates platform, enjoying access to datasets published by a range of customers, including Land Information New ZealandStats NZ, and Manaaki Whenua

How do these organisations get their data — some of which is very large, hugely popular, and regularly updated — onto the Koordinates platform? Some customers use our classic ‘drag-and-drop’ upload process. This is useful for smaller data, or data that isn’t regularly updated. 

For organisations with a large amount of frequently-updated data, though, a manual upload process doesn’t make a lot of sense. Manual uploads don’t scale — and they especially don’t scale for large geospatial datasets. 

To solve this problem, Koordinates supports what we call Data Sources. Supported sources include ArcGIS Rest API endpoints, PostgreSQL databases, Amazon S3 buckets, WFS, and CIFS shares. We also support the related Data Gateway, which you can read about here

As part of new Data Management, we’ve made these sources entirely self-service, meaning you can connect, scan, and import data from supported sources without requiring any manual intervention from the Koordinates team. 

After setting up your source, all you need to do is kick off what we call a scan. During a scan, Koordinates connects to the source and discovers data that could be imported. When Koordinates finds supported data, the platform makes it available for you to check, import, and publish. 

The magic of scans of Data Sources is that they make it much easier to manage data updates. To update data published on Koordinates, all you need to do is click ‘scan’ against your connected source. Koordinates will recognise any changes that have been made to the source's data, and flag updated data with a handy 'update’ badge. This makes it much easier for data managers to identify updates that should be published live. 

Data sources are most often used by organisations – such as government agencies – who own a reasonable amount of data that they want to manage within Koordinates. Smaller users, though, such as teams and individuals, may only have a handful of data layers that they wish to manage or share on Koordinates. What should they do? 

We'll talk about that next week, when we cover how Koordinates handles manual uploads.