UR 1: Data Ingestion

[mvmaltitz]

Expectation

As a user, I require access to the following datasets which I will access via the API:

1. Ground Observations (TAHMO stations) - csv
2. Ground Observations (TAHMO stations) - API
3. CBAM Daily Historical Reanalysis (2012 - 2023)
4. CBAM Daily Historical Reanalysis (Bias-Corrected) (2012-2023)
5. Arable weather stations
6. Laser Disdrometers
7. Radiosonde Observations (Windborne)
8. CBAM Short-Term weather forecast (14-days)
9. Salient Seasonal weather forecast (3-months)
10. ECMWF ERA5 Land
11. CABI PRISE csv
12. CABI PRISE API

Acceptance Criteria

I can view and access the following datasets within the GAP

• Ground Observations (TAHMO stations) - csv
• Ground Observations (TAHMO stations) - API
• CBAM Daily Historical Reanalysis (2012 - 2023)
• CBAM Daily Historical Reanalysis (Bias-Corrected) (2012-2023)
• Arable weather stations
• Laser Disdrometers
• Radiosonde Observations (Windborne)
• CBAM Short-Term weather forecast (14-days)
• Salient Seasonal weather forecast (3-months)
• ECMWF ERA5 Land
• CABI PRISE csv
• CABI PRISE API

Test Criteria

• The datasets below are ingested in the GAP
• Ensure the datasets are stored in the correct format
• Ensure that the correct attributes are stored
• Ensure datasets are updated as per "Updated" below

Data Sources

The following datasets need to be ingested into the GAP platform:

Ground Observations (TAHMO stations) - csv

Attribute	Details
Source:	TAHMO Gap Filled Data (csv)
Version:	2018-2024
Resolution:	300+ stations across East Africa
Attributes:	Attribute table
Format:	Database in the GAP
Updated:	Once off ingestion

Ground Observations (TAHMO stations) - API

Attribute	Details
Source:	TAHMO API
Version:
Resolution:	300+ stations across East Africa
Attributes:	Attribute table
Format:
Updated:

CBAM Daily Historical Reanalysis (2012 - 2023)

Attribute	Details
Source:
Version:	2012-2023
Resolution:	4km²
Attributes:	Attribute table
Format:	NetCDF
Updated:	Once off ingestion

CBAM Daily Historical Reanalysis (Bias-Corrected) (2012-2023)

Attribute	Details
Source:	Bias-corrected Reanalysis
Version:	2012-2023
Resolution:	4km²
Attributes:	Attribute table
Format:	NetCDF
Updated:	Once off ingestion

Arable weather stations

Attribute	Details
Source:	API
Version:
Resolution:
Attributes:	Attribute table
Format:
Updated:

Laser Disdrometers

Attribute	Details
Source:	API
Version:
Resolution:
Attributes:	Attribute table
Format:
Updated:

Radiosonde Observations (Windborne)

Attribute	Details
Source:	Windborne API
Version:
Resolution:
Attributes:	Attribute table
Format:
Updated:

• x, y, z location is updated as the balloon is always moving

CBAM Short-Term weather forecast (14-days)

Attribute	Details
Source:	Tomorrow.io API
Version:
Resolution:	4km²
Attributes:	Attribute table
Format:	ZARR
Updated:	Daily

• Today attributes are stored in the GAP for historical data

Salient Seasonal weather forecast (3-months)

Attribute	Details
Source:	Salient API
Version:	v9
Resolution:	9km²
Attributes:	Attribute table
Format:	ZARR
Updated:

ECMWF ERA5 Land

Attribute	Details
Source:
Version:
Resolution:
Attributes:
Format:
Updated:

CABI PRISE csv

Attribute	Details
Source:	csv
Version:	sent every 2 weeks from CABI
Resolution:
Attributes:
Format:
Updated:

CABI PRISE API

Attribute	Details
Source:
Version:
Resolution:
Attributes:
Format:
Updated:

[mvmaltitz]

Test:

• Ground Observations (TAHMO stations) - csv

• Ground Observations (TAHMO stations) - API

• CBAM Daily Historical Reanalysis (2012 - 2023)
  Ingest CBAM Historical Bias Corrected Data #77

• CBAM Daily Historical Reanalysis (Bias-Corrected) (2012-2023)

• Arable weather stations

• Laser Disdrometers

• Radiosonde Observations (Windborne)

• CBAM Short-Term weather forecast (14-days)
  Replace Tomorrow.io API Call to GAP Zarr File for SPW #118
  Short-term Forecast Data issue #186

• Salient Seasonal weather forecast (3-months)

• ECMWF ERA5 Land
  Create data ingestor: ECMWF #37

• CABI PRISE csv
  Cabi data ingestor #227

• CABI PRISE API
  We do not have access to this yet