Simplify fiboa publish for catalog-driven publication - #214
Open
ivorbosloper wants to merge 74 commits into
Open
Simplify fiboa publish for catalog-driven publication#214ivorbosloper wants to merge 74 commits into
fiboa publish for catalog-driven publication#214ivorbosloper wants to merge 74 commits into
Conversation
Co-authored-by: Matthias Mohr <matthias@mohr.ws>
# Conflicts: # CHANGELOG.md # pixi.lock
…easing memory requirements for large data sets
`fiboa publish` now only converts, validates, builds PMTiles and writes a collection.json with relative links, file:size/file:checksum (multihash), a web-map-links v1.3.0 `pmtiles` link with `pmtiles:layers` and a `visual` asset. README/LICENSE generation, the data-survey lookup and the S3 upload are gone; catalogs such as fieldsoftheworld/harmonized-field-data-catalog own those. spdx-license-list is only needed by tests and moves to the dev feature. Also: - FiboaBaseConverter: keep the determination:datetime column that `use_variant_as_determination` adds; it was removed again as unlisted (affected dk, hr). - be_vlg: drop plots without a crop code (one in 2023 failed validation), take the determination date from the variant year, add the 2026 edition. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ivorbosloper
changed the title
fiboa publish for catalog-driven publication
…note Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
vecorel-cli extracts with plain py7zr, which cannot read .7z.001 splits; the volumes are downloaded as plain files and extracted here once. multivolumefile is a py7zr dependency, so it is already available. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…apping (us_usda_cropland) commons.ec kept an older copy of ec_url/load_ec_mapping that prefixed the EuroCrops base URL to an already absolute https URL; re-export the hcat versions instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… uppercase rename Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ortolan # Conflicts: # pixi.lock # pyproject.toml
…rmination:datetime from issue_year (UTC) The DuckDB converter picked the 'test' variant first when publishing without --variant, and hardcoded determination:datetime 2024 for every year. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
# Conflicts: # fiboa_cli/publish.py # pyproject.toml
- test mocks load_ec_mapping in both commons.ec and commons.hcat (the es converter imports the mixin from hcat directly), with a real-fetch fallback for converters that have no local mapping fixture - add tests/data-files/convert/es/es.csv fixture - relax the HCAT sanity assert to >0 distinct mapped codes (the es fixture is single-crop) - pin vecorel-cli back to 0.2.15 (0.2.16 is unreleased) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Parts only serve resuming a failed run; leaving them next to the merged output gets them uploaded by publish pipelines that glob *.parquet. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…corel-cli vecorel_cli.vecorel.hilbert exists in no released version (per_file.py was written against unreleased 0.2.16); fall back to a fiboa_cli implementation on geopandas' hilbert-curve internals with a per-CRS area-of-use reference frame. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… 2018->2017 (no 2018 archive exists), per-variant determination dates Real archives also carry ILOTS_ANONYMES.gpkg, so **/*.gpkg would match two files; the test pins variant 2022 instead of relying on dict order. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… the cache folder The Koordinates portal needs a login, so the variant sources are the downloaded zips' filenames, resolved against the cache folder (vecorel treats bare local names as cwd-relative and would silently create an empty file there). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nation from the variant year CSB1724.gdb carries a CDL<year> column per year, so every edition reads the same cached source; the old constant 2023-05-01 determination was wrong for anything but the 2023 edition. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ssic ones Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
|
The tests fail. That's not intentional, I guess? |
…dules Covers the new PerFileBaseConverter paths (multi-source merge, part cleanup, re-sort of non-Hilbert parts, error handling), the vendored hilbert module, and the one-line command modules — bringing coverage back over the 80% gate (72.2% -> 81.5%). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ocal mapping fixtures - per_file.py and duckdb.py kept pyarrow readers open across os.remove / os.replace / in-place rewrites; harmless on POSIX, fails on Windows (part cleanup silently no-opped, duckdb 1.1 post-processing silently downgraded). All readers now close deterministically. - commit the 17 crop-mapping CSVs the converter tests previously fetched from the network at test time (a fiboa.org 503 failed the ch test on a Windows runner); tests are now hermetic. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Collaborator
Author
I fixed it. Coverage was too low, and windows handles open files in a different way. Also fixed failed network issues (to fiboa.org) |
…anges Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Old dk editions carry a handful of rows with a null Marknr; the id column is non-nullable, so the write failed after a full conversion. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…daries Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
gpio check flagged published files as spatially unordered. Measured from the published footers: the plain writer yields mediocre locality, and the DuckDB converter's 1-arg ST_Hilbert is meaningless without bounds (jp row groups span 38% of Japan). The per-file merge sort measures excellent (groups 0.1% of extent). One sort implementation now serves every path: _ensure_hilbert_sorted (cheap bbox-only sortedness probe, 50k-row groups) runs at the end of the DuckDB converter and in fiboa publish — re-running publish over an existing parquet doubles as the repair tool for published data. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
take() concatenates chunks; int32 offsets overflow past 2 GB of WKB (us/jp editions). Parquet's BYTE_ARRAY is identical for large_binary. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Simplifies catalog-driven publication while improving large-dataset conversion and updating dataset adapters.
Changes:
- Produces local GeoParquet, PMTiles, and STAC artifacts without uploading.
- Adds DuckDB and per-file Hilbert-sorted conversion pipelines.
- Updates numerous dataset sources, mappings, variants, and tests.
Reviewed changes
Copilot reviewed 65 out of 71 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
tests/test_publish.py |
Tests new publication artifacts. |
tests/test_per_file.py |
Tests streaming merge behavior. |
tests/test_convert.py |
Expands converter coverage. |
tests/test_command_modules.py |
Adds command registration smoke tests. |
tests/data-files/publish/BE-VLG-survey.md |
Removes obsolete survey fixture. |
tests/data-files/convert/us_ca_scm/scm.csv |
Adds SCM mapping fixture. |
tests/data-files/convert/si/si.csv |
Adds Slovenia mapping fixture. |
tests/data-files/convert/se/se.csv |
Adds Sweden mapping fixture. |
tests/data-files/convert/pt/pt.csv |
Adds Portugal mapping fixture. |
tests/data-files/convert/lt/lt_2021.csv |
Adds Lithuania mapping fixture. |
tests/data-files/convert/ie/ie.csv |
Adds Ireland mapping fixture. |
tests/data-files/convert/hr/hr_2020.csv |
Adds Croatia mapping fixture. |
tests/data-files/convert/ec_si/si_2021.csv |
Adds EuroCrops Slovenia fixture. |
tests/data-files/convert/ec_lv/lv_2021.csv |
Adds EuroCrops Latvia fixture. |
tests/data-files/convert/ch/ch.csv |
Adds Switzerland mapping fixture. |
README.md |
Documents simplified publishing. |
pyproject.toml |
Updates runtime and development dependencies. |
fiboa_cli/registry.py |
Registers national Spain converter. |
fiboa_cli/publish.py |
Builds catalog-ready local artifacts. |
fiboa_cli/datasets/us_usda_cropland.py |
Expands USDA variants. |
fiboa_cli/datasets/sk.py |
Corrects Slovakia identifiers. |
fiboa_cli/datasets/nz.py |
Supports manual cached variants. |
fiboa_cli/datasets/nl.py |
Updates Dutch sources and variants. |
fiboa_cli/datasets/jp.py |
Migrates Japan to DuckDB. |
fiboa_cli/datasets/fr.py |
Supports multipart French archives. |
fiboa_cli/datasets/es.py |
Adds national Spain converter. |
fiboa_cli/datasets/es_vc.py |
Uses relocated Spain base. |
fiboa_cli/datasets/es_pv.py |
Uses relocated Spain base. |
fiboa_cli/datasets/es_nc.py |
Uses relocated Spain base. |
fiboa_cli/datasets/es_md.py |
Updates archive matching. |
fiboa_cli/datasets/es_ib.py |
Supports current Balearic service. |
fiboa_cli/datasets/es_ga.py |
Uses relocated Spain base. |
fiboa_cli/datasets/es_ex.py |
Uses relocated Spain base. |
fiboa_cli/datasets/es_cm.py |
Fixes year-specific REST access. |
fiboa_cli/datasets/es_cl.py |
Updates nested archive handling. |
fiboa_cli/datasets/es_cb.py |
Derives determination dates. |
fiboa_cli/datasets/es_cat.py |
Fixes 2024 source handling. |
fiboa_cli/datasets/es_base.py |
Extracts regional Spain base. |
fiboa_cli/datasets/es_ar.py |
Adds municipality-based downloads. |
fiboa_cli/datasets/es_an.py |
Updates mapping and dates. |
fiboa_cli/datasets/ee.py |
Names cached WFS files. |
fiboa_cli/datasets/dk.py |
Handles crop-less editions. |
fiboa_cli/datasets/de_th.py |
Documents INSPIRE source. |
fiboa_cli/datasets/de_bb.py |
Corrects source encoding. |
fiboa_cli/datasets/data-files/es_cl_prv.csv |
Completes province mapping. |
fiboa_cli/datasets/data-files/es_cat.csv |
Adds Catalonian crop names. |
fiboa_cli/datasets/cz.py |
Supports nested shapefiles. |
fiboa_cli/datasets/commons/hcat.py |
Adjusts HCAT mapping behavior. |
fiboa_cli/datasets/commons/euro_land.py |
Handles empty crop codes. |
fiboa_cli/datasets/commons/ec.py |
Consolidates mapping utilities. |
fiboa_cli/datasets/be_vlg.py |
Adds 2026 and date handling. |
fiboa_cli/conversion/per_file.py |
Adds streaming spatial merge. |
fiboa_cli/conversion/hilbert.py |
Adds Hilbert-order helpers. |
fiboa_cli/conversion/fiboa_converter.py |
Preserves dates and filters nulls. |
fiboa_cli/conversion/duckdb.py |
Adds scalable DuckDB conversion. |
fiboa_cli/conversion/converter_rest.py |
Improves REST cache handling. |
CHANGELOG.md |
Records publication and converter changes. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+147
to
+151
| is_current = ( | ||
| stac_file.exists() | ||
| and stac_file.stat().st_mtime >= parquet_file.stat().st_mtime | ||
| and (pmtiles_file is None or stac_file.stat().st_mtime >= pmtiles_file.stat().st_mtime) | ||
| ) |
| [ | ||
| "tippecanoe", | ||
| *tmp_opts, | ||
| *tippecanoe_opts.split(), |
Comment on lines
+88
to
+97
| if isinstance(urls, str): | ||
| sources = f'"{urls}"' | ||
| else: | ||
| paths = [] | ||
| for url in urls: | ||
| if isinstance(url, tuple): | ||
| paths.append(f'"{url[0]}"') | ||
| else: | ||
| paths.append(f'"{url}"') | ||
| sources = "[" + ",".join(paths) + "]" |
Comment on lines
+220
to
+221
| except Exception as e: | ||
| self.warning(f"GeoParquet 1.1 post-processing failed: {e}") |
| except ImportError: | ||
| from .hilbert import crs_total_bounds | ||
|
|
||
| total_bounds = crs_total_bounds(crs) |
Comment on lines
+301
to
+305
| if pa.types.is_binary(f.type): | ||
| fields.append(f.with_type(pa.large_binary())) | ||
| widened = True | ||
| elif pa.types.is_string(f.type): | ||
| fields.append(f.with_type(pa.large_string())) |
Comment on lines
+56
to
+59
| code_sources = [k for k, v in self.columns.items() if v in ("crop:code", "crop:name")] | ||
| if not any(k in gdf.columns for k in code_sources): | ||
| # this edition carries no crop columns at all: nothing to map | ||
| return gdf |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
fiboa publishnow only converts, validates, builds PMTiles and writes acollection.jsonwith relative links,file:size/file:checksum(sha256 multihash), a web-map-links v1.3.0pmtileslink withpmtiles:layers, and avisualasset. README/LICENSE generation, the data-survey lookup and theaws s3 syncupload are removed;--generate-meta,--data-url,--s3-upload-path,--yes,--editor,--converted-by,--data-survey-urlare gone,--no-pmtilesand--tippecanoe-optsare new. README and CHANGELOG updated.spdx-license-listis only used by tests and moves to the pixidevfeature.FiboaBaseConverter: thedetermination:datetimecolumn added byuse_variant_as_determinationwas dropped again by the base converter's "remove unlisted columns" step; it is now listed incolumnsautomatically (affected dk, hr).be_vlg: drop plots without a crop code (one plot in the 2023 edition failed validation), derive the determination date from the variant year instead of a constant, add the 2026 edition.Why
Publishing moves to the git-backed Portolan catalog at https://github.com/fieldsoftheworld/harmonized-field-data-catalog (source.coop/ftw/harmonized-field-data), which owns the catalog layer (styles, thumbnails, AGENTS.md/README, uploads). fiboa-cli stays the single home of conversion logic and produces the per-dataset building blocks.
Verification
The three pilot collections built with this branch pass
rashid check --livewith 0 errors against the published catalog.🤖 Generated with Claude Code