diff --git a/README.md b/README.md index 06c031c..946e299 100644 --- a/README.md +++ b/README.md @@ -12,35 +12,12 @@ For analytic questions like "find the items in the Sentinel-2 collection in June See the [STAC-GeoParquet specification](./spec/stac-geoparquet-spec.md) for details on the exact schema of the written Parquet files. -## Usage - -Use `stac_geoparquet.to_arrow.stac_items_to_arrow` and -`stac_geoparquet.from_arrow.stac_table_to_items` to convert between STAC items -and Arrow tables. Arrow Tables of STAC items can be written to parquet with -`stac_geoparquet.to_parquet.to_parquet`. - -Note that `stac_geoparquet` lifts the keys in the item `properties` up to the top level of the DataFrame, similar to `geopandas.GeoDataFrame.from_features`. - -```python ->>> import requests ->>> import stac_geoparquet.arrow ->>> import pyarrow.parquet ->>> import pyarrow as pa - ->>> items = requests.get( -... "https://planetarycomputer.microsoft.com/api/stac/v1/collections/sentinel-2-l2a/items" -... ).json()["features"] ->>> table = pa.Table.from_batches(stac_geoparquet.arrow.parse_stac_items_to_arrow(items)) ->>> stac_geoparquet.arrow.to_parquet(table, "items.parquet") ->>> table2 = pyarrow.parquet.read_table("items.parquet") ->>> items2 = list(stac_geoparquet.arrow.stac_table_to_items(table2)) -``` - + diff --git a/docs/api/legacy.md b/docs/api/legacy.md index 2084cf6..69c877c 100644 --- a/docs/api/legacy.md +++ b/docs/api/legacy.md @@ -2,6 +2,23 @@ The API listed here was the initial non-Arrow-based STAC-GeoParquet implementation, converting between JSON and GeoPandas directly. For large collections of STAC items, using the new Arrow-based functionality (under the `stac_geoparquet.arrow` namespace) will be more performant. +Note that `stac_geoparquet` lifts the keys in the item `properties` up to the top level of the DataFrame, similar to `geopandas.GeoDataFrame.from_features`. + +```python +>>> import requests +>>> import stac_geoparquet.arrow +>>> import pyarrow.parquet +>>> import pyarrow as pa + +>>> items = requests.get( +... "https://planetarycomputer.microsoft.com/api/stac/v1/collections/sentinel-2-l2a/items" +... ).json()["features"] +>>> table = pa.Table.from_batches(stac_geoparquet.arrow.parse_stac_items_to_arrow(items)) +>>> stac_geoparquet.arrow.to_parquet(table, "items.parquet") +>>> table2 = pyarrow.parquet.read_table("items.parquet") +>>> items2 = list(stac_geoparquet.arrow.stac_table_to_items(table2)) +``` + ::: stac_geoparquet.to_geodataframe ::: stac_geoparquet.to_item_collection ::: stac_geoparquet.to_dict diff --git a/docs/schema.md b/docs/schema.md index 0491aa6..adbf794 100644 --- a/docs/schema.md +++ b/docs/schema.md @@ -1,10 +1,12 @@ # Schema considerations -A _schema_ is a set of named fields that describe what data types exist for each column in a tabular dataset. +A STAC Item is a JSON object to describe an external geospatial dataset. The STAC specification defines a common core, plus a variety of extensions. Additionally, STAC Items may include custom extensions outside the common ones. Crucially, the majority of the specified fields in the core spec and extensions define optional keys. Those keys often differ across STAC collections and may even differ within a single collection across items. -JSON and Arrow/Parquet have opposite expectations around schemas. JSON is fully schemaless; every object in a JSON array may have a completely different schema. Meanwhile Arrow and Parquet require a strict schema upfront, and it is impossible for a row in an Arrow table or Parquet file to not comply with the upfront schema. Trying to write STAC data with mixed schemas to Parquet can easily produce exceptions. +STAC's flexibility is a blessing and a curse. The flexibility of schemaless JSON allows for very easy writing as each object can be dumped separately to JSON. Every item is allowed to have a different schema. And newer items are free to have a different schema than older items in the same collection. But this write-time flexibility makes it harder to read as there are no guarantees (outside STAC's few required fields) about what fields exist. -Because of these opposite expectations, schema inference is the most difficult part of converting STAC from JSON to GeoParquet. +Parquet is the complete opposite of JSON. Parquet has a strict schema that must be known before writing can start. This puts the burden of work onto the writer instead of the reader. Reading Parquet is very efficient because the file's metadata defines the exact schema of every record. This also enables use cases like reading specific columns that would not be possible without a strict schema. + +This conversion from schemaless to strict-schema is the difficult part of converting STAC from JSON to GeoParquet, especially for large input datasets like STAC that are often larger than memory. ## Full scan over input data @@ -16,7 +18,11 @@ This is time consuming as it requires two full passes over the input data: once Alternatively, the user can pass in an Arrow schema themselves using the `schema` parameter of [`parse_stac_ndjson_to_arrow`][stac_geoparquet.arrow.parse_stac_ndjson_to_arrow]. This `schema` must match the on-disk schema of the the STAC JSON data. -## Merging data with schema mismatch +## Multiple schemas per collection + +It is also possible to write multiple Parquet files with STAC data where each Parquet file may have a different schema. This simplifies the conversion and writing process but makes reading and using the Parquet data harder. + +### Merging data with schema mismatch If you've created STAC GeoParquet data where the schema has updated, you can use [`pyarrow.concat_tables`][pyarrow.concat_tables] with `promote_options="permissive"` to combine multiple STAC GeoParquet files. diff --git a/docs/usage.md b/docs/usage.md index 8f04b05..746b9be 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -1 +1,50 @@ # Usage + +Except for the [legacy API](api/legacy.md), [Apache Arrow](https://arrow.apache.org/) is used as the in-memory interchange format between all formats. While some end-to-end helper functions are provided, the user can go through Arrow objects for maximal flexibility in the conversion process. + +All functionality that goes through Arrow is currently exported via the `stac_geoparquet.arrow` namespace. + +## `dict`/JSON - Arrow conversion + +### Convert `dict`s to Arrow + +Use [`parse_stac_items_to_arrow`][stac_geoparquet.arrow.parse_stac_items_to_arrow] to convert STAC items either in memory or on disk to a stream of Arrow record batches. This accepts either an iterable of Python `dict`s or an iterable of [`pystac.Item`][pystac.Item] objects. + +### Convert JSON to Arrow + +[`parse_stac_ndjson_to_arrow`][stac_geoparquet.arrow.parse_stac_ndjson_to_arrow] is a helper function to take one or more JSON or newline-delimited JSON files on disk, infer the schema from all of them, and convert the data to a stream of Arrow record batches. + +### Convert Arrow to `dict`s + +Use [`stac_table_to_items`][stac_geoparquet.arrow.stac_table_to_items] to convert a table or stream of Arrow record batches of STAC data to a generator of Python `dict`s. This accepts either a `pyarrow.Table` or a `pyarrow.RecordBatchReader`, which allows conversions of larger-than-memory files in a streaming manner. + +### Convert Arrow to JSON + +Use [`stac_table_to_ndjson`][stac_geoparquet.arrow.stac_table_to_ndjson] to convert a table or stream of Arrow record batches of STAC data to a generator of Python `dict`s. This accepts either a `pyarrow.Table` or a `pyarrow.RecordBatchReader`, which allows conversions of larger-than-memory files in a streaming manner. + +## Parquet + +Use [`to_parquet`][stac_geoparquet.arrow.to_parquet] to write STAC Arrow data in memory. This is a special function to ensure that [GeoParquet](https://geoparquet.org/) 1.0 or 1.1 metadata is written to the Parquet file. + +[`parse_stac_ndjson_to_parquet`][stac_geoparquet.arrow.parse_stac_ndjson_to_parquet] is a helper that connects reading (newline-delimited) JSON on disk to writing out to a Parquet file. + +No special API is required for reading a STAC GeoParquet file back into Arrow. You can use [`pyarrow.parquet.read_table`][pyarrow.parquet.read_table] or [`pyarrow.parquet.ParquetFile`][pyarrow.parquet.ParquetFile] directly to read the STAC GeoParquet data back into Arrow. + +## Delta Lake + + +Use [`parse_stac_ndjson_to_delta_lake`][stac_geoparquet.arrow.parse_stac_ndjson_to_delta_lake] to read (newline-delimited) JSON on disk and write out to a Delta Lake table. + +No special API is required for reading a STAC Delta Lake table back into Arrow. You can use the [`DeltaTable`][deltalake.DeltaTable] class directly to read the data back into Arrow. + +!!! important + Arrow has a null data type, where every value in the column is always null, but Delta Lake does not. This means that for any column inferred to have a `null` data type, writing to Delta Lake will error with + ``` + _internal.SchemaMismatchError: Invalid data type for Delta Lake: Null + ``` + + This is a problem because if all items in a STAC Collection have a `null` JSON key, it gets inferred as an Arrow `null` type. For example, in the `3dep-lidar-copc` collection in the tests, it has `start_datetime` and `end_datetime` fields, and so according to the spec, `datetime` is always `null`. This column would need to be casted to a timestamp type before being written to Delta Lake. + + This means we cannot write this collection to Delta Lake **solely with automatic schema inference**. + + In such cases, users may need to manually update the inferred schema to cast any `null` type to another Delta Lake-compatible type.