Open Source GIS stacks

[metazool]

Identify places where current data and services publication workflows could hugely benefit from adopting an open standard, open source GIS stack. In either of these cases:

1. Improved data re-usability and unexpected applications, both internal and external to CEH
2. Pushing at the edges of proprietary / platform options in ways that are expensive, restrictive or both

The first case is around the Land Cover Maps specifically - straining at publishing massive rasters through EIDC. There are probably other projects, looking towards Biodiversity, that have either similar requirements or an existing more mature approach which could be wider adopted.

Heavier weight (raster+) data

• [pangeo-forge](https://github.com/NERC-CEH/rse_group/blob/main/infrastructure/pipelines/pangeo_forge.md) for ETL of collections of very large rasters to Cloud Optimised GeoTIFF in cloud storage
• STAC catalog adoption - see the pangeo-forge STAC catalog or MS Planetary Computer. Should this be part of a wider effort across NERC data centres (is it core competency)? - suggested backend is stac-fastapi-pgstac - what do internal support arrangements for PostGIS (there are some) look like?
• See also object_store_tutorial for prior art by @mattjbr123 on pangeo-forge for gridded data. There is STAC/Zarr and there's a Cloud Optimised Point Cloud community standard underpinned by work in the PDAL library - it could be the same picture

Lighter weight (vector+) data

• Less clear what the default stack is. For serving vector tiles, is it Tegola? Geoserver? What's already well supported either by EDS or with transitional arrangements to central support?
• tippecanoe - commercially supported fork of the original Mapbox library for vector tile generation
• What other use cases (I've seen two, one in Atmospheric Chemistry and one in Hydrology, for processing large vectors into smaller tiles for use within external-facing analysis portal (policy stakeholders, etc). What's the current cost of the proprietary cloud platform, plus expected and suppressed usage? What options are there for off the shelf pipelines with a good level of metadata recording that we could just deploy somewhere?

[metazool]

Could and should have added a lot more detail here but it's better to write something up for future reference than leave it until all the thinking's done!

Thread originally prompted in late June on reading Accessing Planetary Computer STAC files in DuckDB by @robintw

[robintw]

Thanks for the reference to my blog post @metazool.

I'd be happy to chat about this sometime - you may be aware that I do freelance work in this area (see http://www.rtwilson.com). I would strongly in favour of you using modern cloud-native technologies like STAC, COG, PostGIS and so on.