Feature/issue 198 #207

nikki-t · 2024-07-19T19:46:12Z

Github Issue: #198

Description

Implement queries for CMR and Hydrocron.

Define a Lambda function to track granule ingestion into Hydrocron and populate the program with the appropriate queries. See #71 for context of larger program.

Overview of work done

Create track_ingest.py with a Track class to keep track of granules ingested into Hydrocron.
Query CMR for granules in a determined temporal range by revision_date query parameter.
Query Hydrocron for granules located in CMR.
Create placeholder definitions for other operations.
Define unit tests to test functionality.

Overview of verification done

Created three new unit tests

Test query to CMR (uses vcrpy to capture CMR API response)
Test query to Hydrocron
Test specific Hydrocron query by granuleUR index

New and existing unit tests pass.

Overview of integration done

Deployed to SIT environment and ran tested the track ingest lambda on the following event:

{
  "collection_shortname": "SWOT_L2_HR_RiverSP_reach_2.0",
  "collection_start_date": "20221216",
  "hydrocron_table": "hydrocron-swot-reach-table"
}

Logged results:

2024-07-19T19:30:40.616Z [INFO] 2024-07-19T19:30:40.616Z  Event: {'collection_shortname': 'SWOT_L2_HR_RiverSP_reach_2.0', 'collection_start_date': '20221216', 'hydrocron_table': 'hydrocron-swot-reach-table'}
2024-07-19T19:30:40.619Z [INFO] 2024-07-19T19:30:40.618Z  Search URL: https://cmr.earthdata.nasa.gov/search/granules.umm_json
2024-07-19T19:30:40.619Z [INFO] 2024-07-19T19:30:40.619Z  Search Parameters: {'short_name': 'SWOT_L2_HR_RiverSP_reach_2.0', 'revision_date': '2022-12-16T00:00:00Z,2024-07-19T19:30:40Z', 'page_size': 2000}
2024-07-19T19:30:46.259Z [INFO] 2024-07-19T19:30:46.259Z  Searching for more results...2000 out of 11651
2024-07-19T19:30:50.904Z [INFO] 2024-07-19T19:30:50.904Z  Searching for more results...4000 out of 11651
2024-07-19T19:30:56.184Z [INFO] 2024-07-19T19:30:56.184Z  Searching for more results...6000 out of 11651
2024-07-19T19:31:01.241Z [INFO] 2024-07-19T19:31:01.241Z  Searching for more results...8000 out of 11651
2024-07-19T19:31:05.758Z [INFO] 2024-07-19T19:31:05.758Z  Searching for more results...10000 out of 11651
2024-07-19T19:31:09.541Z [INFO] 2024-07-19T19:31:09.540Z  Searching for more results...11651 out of 11651
2024-07-19T19:31:09.761Z [INFO] 2024-07-19T19:31:09.761Z  Located 11651 granules in CMR.
2024-07-19T19:31:09.862Z [INFO] 2024-07-19T19:31:09.862Z  Found credentials in environment variables.
2024-07-19T19:35:27.259Z [INFO] 2024-07-19T19:35:27.259Z  Located 546 granules in Hydrocron.
2024-07-19T19:35:27.259Z [INFO] 2024-07-19T19:35:27.259Z  Elapsed: 0:04:46.643316

PR checklist:

Linted
Updated unit tests
Updated changelog
Integration testing

See Pull Request Review Checklist for pointers on reviewing this pull request

- Includes placeholders for other operations needed to track granule ingest. - GranuleUR query for Hydrocron tables.

- Test CMR and hydrocron queries - Test granuleUR query - Update database to include granuleUR GSI

CHANGELOG.md

hydrocron/db/track_ingest.py

frankinspace · 2024-07-19T19:57:14Z

hydrocron/db/track_ingest.py

+        self.hydrocron_granules = {}
+        self.hydrocron_table = hydrocron_table
+        self.revision_start = self._get_revision_start(collection_start_date)
+        self.revision_end = datetime.datetime.now(timezone.utc)  # TODO - Decide on latency and subtract from current datetime


wrt the TODO. @torimcd do we have any metrics on latency between observation time and archival time?

very sorry for the delay on this, for fwd stream the latency from observation to archival is 48 - 72 hours. In the case of a reprocessing campaign I don't think we'd want to calculate from observation time though, since that could be months/years. The nominal latency from time between CNM notification and successful ingest is ~3 min, which will be true for both forward and reprocessed granules

Maybe we can set the latency to 1 hour and query from the calculated revision start date to the current time minus 1 hour? I will plan on this for now and we can always adjust - I actually might make the latency into an environment variable that we can adjust when needed. I will include this work as a part of #211.

And now I will merge this PR as I think all work is complete!

hydrocron/db/track_ingest.py

terraform/hydrocron-lambda.tf

tests/conftest.py

nikki-t · 2024-07-24T14:16:44Z

Implemented cmr_python query to replace CMR API query.

Ran test on entire reach collection in SIT:

11,981 granules found in CMR.
546 granules found in Hydrocron.
Execution time: 00:04:51.669945

Ran test on entire node collection in SIT:

11,982 granules found in CMR.
566 grnaules found in Hydrocron.
Execution time: 00:03:42.835959

I think this PR can be merged and closed after we figure out the latency between observation and archival time.

* /version 1.3.0a0 * Update build.yml * /version 1.3.0a1 * /version 1.3.0a2 * Feature/issue 175 - Update docs to point to OPS (#176) * changelog * update examples, remove load_data readme, info moved to wiki * Dependency update to fix snyk scan * issues/101: Support for HTTP Accept header (#172) * Reorganize timeseries code to prep for Accept header * Enable Accept header to return response of specific content-type * Fix whitespace and string continuation * Make error handling consistent and add an additional test where a reach can't be found * Update changelog with issue for unreleased version * Add 415 status code to API definition * Few minor cleanup items * Few minor cleanup items * Update to aiohttp@3.9.4 * Fix dependencies --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.3.0a3 * issues/102: Support compression of API response (#173) * Enable payload compression * Update changelog with issue --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.3.0a4 * Feature/issue 100 Add option to 'compact' GeoJSON result into single feature (#177) * Reorganize timeseries code to prep for Accept header * Enable Accept header to return response of specific content-type * Fix whitespace and string continuation * Make error handling consistent and add an additional test where a reach can't be found * Update changelog with issue for unreleased version * Add 415 status code to API definition * Few minor cleanup items * Few minor cleanup items * Update to aiohttp@3.9.4 * Fix dependencies * Update required query parameters based on current API functionality * Enable return of 'compact' GeoJSON response * Fix linting and add test data * Update documentation for API accept headers and compact GeoJSON response * Fix references to incorrect Accept header examples --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.3.0a5 * Feature/issue 183 (#185) * Provide introduction to timeseries endpoint * Remove _units in fields list * Fix typo * Update examples with Accept headers and compact query parameter * Add issue to changelog * Fix typo in timeseries documentation * Update pymysql * Update pymysql * Provide clarity on accept headers and request parameter fields * /version 1.3.0a6 * Feature/issue 186 Implement API keys (#188) * API Gateway Lambda authorizer to facilitate API keys and usage plans * Unit tests to test Lambda authorizer * Fix terraform file formatting * API Gateway Lambda Authorizer - Lambda function - API Keys and Authorizer definition in OpenAPI spec - API gateway API keys - API gateway usage plans - SSM parameters for API keys * Fix trailing whitespace * Set default region environment variable * Fix SNYK vulnerabilities * Add issue to changelog * Implement custom trusted partner header x-hydrocron-key * Update cryptography for SNYK vulnerability * Update documentation to include API key usage * Update quota and throttle settings for API Gateway * Update API keys documentation to indicate to be implemented * Move API key lookup to Lambda INIT * Remove API key authentication and update API key to x-hydrocron-key * /version 1.3.0a7 * Update changelog for 1.3.0 release * /version 1.4.0a0 * Feature/issue 198 (#207) * Update pylint to deal with errors and fix collection reference * Initial CMR and Hydrocron queries - Includes placeholders for other operations needed to track granule ingest. - GranuleUR query for Hydrocron tables. * Add and set up vcrpy for testing CMR API query * Test track ingest operations - Test CMR and hydrocron queries - Test granuleUR query - Update database to include granuleUR GSI * Update to use track_ingest naming consistently * Initial Lambda function and IAM role definition * Replace deprecated path function with as_file * Add SSM read IAM permissions * Add DynamoDB read permissions * Update track ingest lambda memory * Remove duplicate IAM permissions * Add in permissions to query index * Update changelog * Update changelog description * Use python_cmr for CMR API queries * /version 1.4.0a1 * Add doi to documentation pages (#216) * Update intro.md with DOI * Update overview.md with DOI * /version 1.4.0a2 * issue-193: Add Dynamo DB Table for SWOT Prior Lakes (#209) * add code to handle prior lakes shapefiles, add test prior lake data * update terraform to add prior lake table * fix tests, change to smaller test data file, changelog * linting * reconfigure main load_data method to make more readable and pass linting * lint * lint * fix string casting to lower storage req & update test responses to handle different rounding pattern in coords * update load benchmarking function for linting and add unit test * try parent collection for lakes * update version parsing for parent collection * fix case error * fix lake id reference * add logging to troubleshoot too large features * add item size logging and remove error raise for batch write * clean up logging statements & move numeric_columns assignment * update batch logging statement * Rename constant * Fix temp dir security risk https://rules.sonarsource.com/python/RSPEC-5443/ * Fix temp dir security risk https://rules.sonarsource.com/python/RSPEC-5443/ * fix code coverage calculation --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.4.0a3 * Feature/issue 201 Create a table for tracking granule ingest status (#214) * Define track ingest database and IAM permissions * Update changelog with issue * Modify table structure to support sparse status index * Updated to only apply PITR in ops --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.4.0a4 * Feature/issue 210 - Load large geometry polygons (#219) * add functions to handle null geometries and convert polygons to points * update doi in docs * fix fill null geometries * fix tests and update changelog * /version 1.4.0a5 * Feature/issue 222 - Add granule info to track ingest table on load (#223) * adjust lambdas to populate track ingest table on granule load * changelog * remove test cnm * lint * change error caught when handling checksum * update lambda role permissions to write to track ingest table * fix typo on lake table terraform * set default fill values for checksum and rev date in track status * fix checksum handling in bulk load data * lint * add logging to debug * /version 1.4.0a6 * Feature/issue-225: Create one track ingest table per feature type (#226) * add track ingest tables for each feature type and adjust load data to populate * changelog * /version 1.4.0a7 * Feature/issue 196 Add new feature type to query the API for lake data (#224) * Initial API queries for lake data * Unit tests for lake data * Updates after center point calculations - Removed temp code to calculate a point in API - Implemented unit test to test lake data retrieval - Updated fixtures to load in lake data for testing * Add read lake table permissions to lambda timeseries and track ingest roles * Update documenation to include lake data * Updated documentation to include info on lake centerpoints --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.4.0a8 * Feature/issue 205 - Add Confluence API key (#221) * Fix possible variable references before value is assigned * Define Confluence API key and trusted partner plan limits * Define a list of trusted partner keys and store under single parameter * Define API keys as encrypted envrionment variables for Lambda authorizer * Update authorizer and connection class to use KMS to retrieve API keys * Hack to force lambda deployment when ssm value changes (#218) * Add replace_triggered_by to hydrocron_lambda_authorizer * Introduce environment variable that contains random id which will change whenever an API key value changes. This will force lambda to publish new version of the function. * Remove unnecessary hash function * Update to SSM parameter API key storage and null_resource enviroment variable * Update Terraform and AWS provider * Update API key documentation * Set source_code_hash to force deployment of new image * Downgrade AWS provider to 4.0 to remove inline policy errors * Update docs/timeseries.md --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.4.0a9 * /version 1.4.0a10 * changelog for 1.4.0 release * update dependencies for 1.4.0 release * fix CMR query in UAT * /version 1.4.0rc1 * fix typo in load_data lambda * /version 1.4.0rc2 * fix index on rev date in load data lambda * update dependencies * lint readme * /version 1.4.0rc3 * /version 1.4.0rc4 * fix cmr env search by venue * /version 1.4.0rc5 --------- Co-authored-by: nikki-t <nikki-t@users.noreply.github.com> Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> Co-authored-by: frankinspace <frankinspace@users.noreply.github.com> Co-authored-by: Nikki Tebaldi <17799906+nikki-t@users.noreply.github.com> Co-authored-by: Cassie Nickles <44206543+cassienickles@users.noreply.github.com> Co-authored-by: cassienickles <cassienickles@users.noreply.github.com> Co-authored-by: podaac-cicd[bot] <podaac-cicd[bot]@users.noreply.github.com> Co-authored-by: torimcd <torimcd@users.noreply.github.com>

…gest" status (#227) * /version 1.3.0a0 * Update build.yml * /version 1.3.0a1 * /version 1.3.0a2 * Feature/issue 175 - Update docs to point to OPS (#176) * changelog * update examples, remove load_data readme, info moved to wiki * Dependency update to fix snyk scan * issues/101: Support for HTTP Accept header (#172) * Reorganize timeseries code to prep for Accept header * Enable Accept header to return response of specific content-type * Fix whitespace and string continuation * Make error handling consistent and add an additional test where a reach can't be found * Update changelog with issue for unreleased version * Add 415 status code to API definition * Few minor cleanup items * Few minor cleanup items * Update to aiohttp@3.9.4 * Fix dependencies --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.3.0a3 * issues/102: Support compression of API response (#173) * Enable payload compression * Update changelog with issue --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.3.0a4 * Feature/issue 100 Add option to 'compact' GeoJSON result into single feature (#177) * Reorganize timeseries code to prep for Accept header * Enable Accept header to return response of specific content-type * Fix whitespace and string continuation * Make error handling consistent and add an additional test where a reach can't be found * Update changelog with issue for unreleased version * Add 415 status code to API definition * Few minor cleanup items * Few minor cleanup items * Update to aiohttp@3.9.4 * Fix dependencies * Update required query parameters based on current API functionality * Enable return of 'compact' GeoJSON response * Fix linting and add test data * Update documentation for API accept headers and compact GeoJSON response * Fix references to incorrect Accept header examples --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.3.0a5 * Feature/issue 183 (#185) * Provide introduction to timeseries endpoint * Remove _units in fields list * Fix typo * Update examples with Accept headers and compact query parameter * Add issue to changelog * Fix typo in timeseries documentation * Update pymysql * Update pymysql * Provide clarity on accept headers and request parameter fields * /version 1.3.0a6 * Feature/issue 186 Implement API keys (#188) * API Gateway Lambda authorizer to facilitate API keys and usage plans * Unit tests to test Lambda authorizer * Fix terraform file formatting * API Gateway Lambda Authorizer - Lambda function - API Keys and Authorizer definition in OpenAPI spec - API gateway API keys - API gateway usage plans - SSM parameters for API keys * Fix trailing whitespace * Set default region environment variable * Fix SNYK vulnerabilities * Add issue to changelog * Implement custom trusted partner header x-hydrocron-key * Update cryptography for SNYK vulnerability * Update documentation to include API key usage * Update quota and throttle settings for API Gateway * Update API keys documentation to indicate to be implemented * Move API key lookup to Lambda INIT * Remove API key authentication and update API key to x-hydrocron-key * /version 1.3.0a7 * Update changelog for 1.3.0 release * /version 1.4.0a0 * Feature/issue 198 (#207) * Update pylint to deal with errors and fix collection reference * Initial CMR and Hydrocron queries - Includes placeholders for other operations needed to track granule ingest. - GranuleUR query for Hydrocron tables. * Add and set up vcrpy for testing CMR API query * Test track ingest operations - Test CMR and hydrocron queries - Test granuleUR query - Update database to include granuleUR GSI * Update to use track_ingest naming consistently * Initial Lambda function and IAM role definition * Replace deprecated path function with as_file * Add SSM read IAM permissions * Add DynamoDB read permissions * Update track ingest lambda memory * Remove duplicate IAM permissions * Add in permissions to query index * Update changelog * Update changelog description * Use python_cmr for CMR API queries * /version 1.4.0a1 * Add doi to documentation pages (#216) * Update intro.md with DOI * Update overview.md with DOI * /version 1.4.0a2 * issue-193: Add Dynamo DB Table for SWOT Prior Lakes (#209) * add code to handle prior lakes shapefiles, add test prior lake data * update terraform to add prior lake table * fix tests, change to smaller test data file, changelog * linting * reconfigure main load_data method to make more readable and pass linting * lint * lint * fix string casting to lower storage req & update test responses to handle different rounding pattern in coords * update load benchmarking function for linting and add unit test * try parent collection for lakes * update version parsing for parent collection * fix case error * fix lake id reference * add logging to troubleshoot too large features * add item size logging and remove error raise for batch write * clean up logging statements & move numeric_columns assignment * update batch logging statement * Rename constant * Fix temp dir security risk https://rules.sonarsource.com/python/RSPEC-5443/ * Fix temp dir security risk https://rules.sonarsource.com/python/RSPEC-5443/ * fix code coverage calculation --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.4.0a3 * Feature/issue 201 Create a table for tracking granule ingest status (#214) * Define track ingest database and IAM permissions * Update changelog with issue * Modify table structure to support sparse status index * Updated to only apply PITR in ops --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.4.0a4 * Feature/issue 210 - Load large geometry polygons (#219) * add functions to handle null geometries and convert polygons to points * update doi in docs * fix fill null geometries * fix tests and update changelog * /version 1.4.0a5 * Feature/issue 222 - Add granule info to track ingest table on load (#223) * adjust lambdas to populate track ingest table on granule load * changelog * remove test cnm * lint * change error caught when handling checksum * update lambda role permissions to write to track ingest table * fix typo on lake table terraform * set default fill values for checksum and rev date in track status * fix checksum handling in bulk load data * lint * add logging to debug * /version 1.4.0a6 * Add SSM parameter read for last run time * Feature/issue-225: Create one track ingest table per feature type (#226) * add track ingest tables for each feature type and adjust load data to populate * changelog * /version 1.4.0a7 * Feature/issue 196 Add new feature type to query the API for lake data (#224) * Initial API queries for lake data * Unit tests for lake data * Updates after center point calculations - Removed temp code to calculate a point in API - Implemented unit test to test lake data retrieval - Updated fixtures to load in lake data for testing * Add read lake table permissions to lambda timeseries and track ingest roles * Update documenation to include lake data * Updated documentation to include info on lake centerpoints --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.4.0a8 * Feature/issue 205 - Add Confluence API key (#221) * Fix possible variable references before value is assigned * Define Confluence API key and trusted partner plan limits * Define a list of trusted partner keys and store under single parameter * Define API keys as encrypted envrionment variables for Lambda authorizer * Update authorizer and connection class to use KMS to retrieve API keys * Hack to force lambda deployment when ssm value changes (#218) * Add replace_triggered_by to hydrocron_lambda_authorizer * Introduce environment variable that contains random id which will change whenever an API key value changes. This will force lambda to publish new version of the function. * Remove unnecessary hash function * Update to SSM parameter API key storage and null_resource enviroment variable * Update Terraform and AWS provider * Update API key documentation * Set source_code_hash to force deployment of new image * Downgrade AWS provider to 4.0 to remove inline policy errors * Update docs/timeseries.md --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.4.0a9 * /version 1.4.0a10 * changelog for 1.4.0 release * /version 1.5.0a0 * Initial track ingest table query * Fix linting and code style * Implement feature count operations * Enable S3 permissions and set environment variable for track lambda * Fix trailing white spaces and code format * Update docstrings for class methods * Implement run time storage in SSM * Query track table unit tests * Update CHANGELOG with issue * Update SSM run time parameter * Fix trailing whitespace * Fix reference to IAM policy * Enable specification of temporal range to search revision date by * Fix SSM put parameter policy * Update IAM permissions for reading track ingest * Enable full temporal search on CMR granules * Add capability to download shapefile granule to count features * Update granule UR to include .zip * Count features via Hydrocron table query * Remove unnecessary s3 permissions * Remove whitespace from blank line * Update cryptography to 43.0.1 * update dependencies * upgrade geopandas * update dependencies --------- Co-authored-by: nikki-t <nikki-t@users.noreply.github.com> Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> Co-authored-by: frankinspace <frankinspace@users.noreply.github.com> Co-authored-by: Victoria McDonald <49625194+torimcd@users.noreply.github.com> Co-authored-by: Cassie Nickles <44206543+cassienickles@users.noreply.github.com> Co-authored-by: cassienickles <cassienickles@users.noreply.github.com> Co-authored-by: podaac-cicd[bot] <podaac-cicd[bot]@users.noreply.github.com> Co-authored-by: Victoria McDonald <victoria.mcdonald@jpl.nasa.gov> Co-authored-by: torimcd <torimcd@users.noreply.github.com>

* /version 1.3.0a0 * Update build.yml * /version 1.3.0a1 * /version 1.3.0a2 * Feature/issue 175 - Update docs to point to OPS (#176) * changelog * update examples, remove load_data readme, info moved to wiki * Dependency update to fix snyk scan * issues/101: Support for HTTP Accept header (#172) * Reorganize timeseries code to prep for Accept header * Enable Accept header to return response of specific content-type * Fix whitespace and string continuation * Make error handling consistent and add an additional test where a reach can't be found * Update changelog with issue for unreleased version * Add 415 status code to API definition * Few minor cleanup items * Few minor cleanup items * Update to aiohttp@3.9.4 * Fix dependencies --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.3.0a3 * issues/102: Support compression of API response (#173) * Enable payload compression * Update changelog with issue --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.3.0a4 * Feature/issue 100 Add option to 'compact' GeoJSON result into single feature (#177) * Reorganize timeseries code to prep for Accept header * Enable Accept header to return response of specific content-type * Fix whitespace and string continuation * Make error handling consistent and add an additional test where a reach can't be found * Update changelog with issue for unreleased version * Add 415 status code to API definition * Few minor cleanup items * Few minor cleanup items * Update to aiohttp@3.9.4 * Fix dependencies * Update required query parameters based on current API functionality * Enable return of 'compact' GeoJSON response * Fix linting and add test data * Update documentation for API accept headers and compact GeoJSON response * Fix references to incorrect Accept header examples --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.3.0a5 * Feature/issue 183 (#185) * Provide introduction to timeseries endpoint * Remove _units in fields list * Fix typo * Update examples with Accept headers and compact query parameter * Add issue to changelog * Fix typo in timeseries documentation * Update pymysql * Update pymysql * Provide clarity on accept headers and request parameter fields * /version 1.3.0a6 * Feature/issue 186 Implement API keys (#188) * API Gateway Lambda authorizer to facilitate API keys and usage plans * Unit tests to test Lambda authorizer * Fix terraform file formatting * API Gateway Lambda Authorizer - Lambda function - API Keys and Authorizer definition in OpenAPI spec - API gateway API keys - API gateway usage plans - SSM parameters for API keys * Fix trailing whitespace * Set default region environment variable * Fix SNYK vulnerabilities * Add issue to changelog * Implement custom trusted partner header x-hydrocron-key * Update cryptography for SNYK vulnerability * Update documentation to include API key usage * Update quota and throttle settings for API Gateway * Update API keys documentation to indicate to be implemented * Move API key lookup to Lambda INIT * Remove API key authentication and update API key to x-hydrocron-key * /version 1.3.0a7 * Update changelog for 1.3.0 release * /version 1.4.0a0 * Feature/issue 198 (#207) * Update pylint to deal with errors and fix collection reference * Initial CMR and Hydrocron queries - Includes placeholders for other operations needed to track granule ingest. - GranuleUR query for Hydrocron tables. * Add and set up vcrpy for testing CMR API query * Test track ingest operations - Test CMR and hydrocron queries - Test granuleUR query - Update database to include granuleUR GSI * Update to use track_ingest naming consistently * Initial Lambda function and IAM role definition * Replace deprecated path function with as_file * Add SSM read IAM permissions * Add DynamoDB read permissions * Update track ingest lambda memory * Remove duplicate IAM permissions * Add in permissions to query index * Update changelog * Update changelog description * Use python_cmr for CMR API queries * /version 1.4.0a1 * Add doi to documentation pages (#216) * Update intro.md with DOI * Update overview.md with DOI * /version 1.4.0a2 * issue-193: Add Dynamo DB Table for SWOT Prior Lakes (#209) * add code to handle prior lakes shapefiles, add test prior lake data * update terraform to add prior lake table * fix tests, change to smaller test data file, changelog * linting * reconfigure main load_data method to make more readable and pass linting * lint * lint * fix string casting to lower storage req & update test responses to handle different rounding pattern in coords * update load benchmarking function for linting and add unit test * try parent collection for lakes * update version parsing for parent collection * fix case error * fix lake id reference * add logging to troubleshoot too large features * add item size logging and remove error raise for batch write * clean up logging statements & move numeric_columns assignment * update batch logging statement * Rename constant * Fix temp dir security risk https://rules.sonarsource.com/python/RSPEC-5443/ * Fix temp dir security risk https://rules.sonarsource.com/python/RSPEC-5443/ * fix code coverage calculation --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.4.0a3 * Feature/issue 201 Create a table for tracking granule ingest status (#214) * Define track ingest database and IAM permissions * Update changelog with issue * Modify table structure to support sparse status index * Updated to only apply PITR in ops --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.4.0a4 * Feature/issue 210 - Load large geometry polygons (#219) * add functions to handle null geometries and convert polygons to points * update doi in docs * fix fill null geometries * fix tests and update changelog * /version 1.4.0a5 * Feature/issue 222 - Add granule info to track ingest table on load (#223) * adjust lambdas to populate track ingest table on granule load * changelog * remove test cnm * lint * change error caught when handling checksum * update lambda role permissions to write to track ingest table * fix typo on lake table terraform * set default fill values for checksum and rev date in track status * fix checksum handling in bulk load data * lint * add logging to debug * /version 1.4.0a6 * Add SSM parameter read for last run time * Feature/issue-225: Create one track ingest table per feature type (#226) * add track ingest tables for each feature type and adjust load data to populate * changelog * /version 1.4.0a7 * Feature/issue 196 Add new feature type to query the API for lake data (#224) * Initial API queries for lake data * Unit tests for lake data * Updates after center point calculations - Removed temp code to calculate a point in API - Implemented unit test to test lake data retrieval - Updated fixtures to load in lake data for testing * Add read lake table permissions to lambda timeseries and track ingest roles * Update documenation to include lake data * Updated documentation to include info on lake centerpoints --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.4.0a8 * Feature/issue 205 - Add Confluence API key (#221) * Fix possible variable references before value is assigned * Define Confluence API key and trusted partner plan limits * Define a list of trusted partner keys and store under single parameter * Define API keys as encrypted envrionment variables for Lambda authorizer * Update authorizer and connection class to use KMS to retrieve API keys * Hack to force lambda deployment when ssm value changes (#218) * Add replace_triggered_by to hydrocron_lambda_authorizer * Introduce environment variable that contains random id which will change whenever an API key value changes. This will force lambda to publish new version of the function. * Remove unnecessary hash function * Update to SSM parameter API key storage and null_resource enviroment variable * Update Terraform and AWS provider * Update API key documentation * Set source_code_hash to force deployment of new image * Downgrade AWS provider to 4.0 to remove inline policy errors * Update docs/timeseries.md --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.4.0a9 * /version 1.4.0a10 * changelog for 1.4.0 release * /version 1.5.0a0 * Initial track ingest table query * Fix linting and code style * Implement feature count operations * Enable S3 permissions and set environment variable for track lambda * Fix trailing white spaces and code format * Update docstrings for class methods * Implement run time storage in SSM * Query track table unit tests * Update CHANGELOG with issue * Update SSM run time parameter * Fix trailing whitespace * Fix reference to IAM policy * Enable specification of temporal range to search revision date by * Fix SSM put parameter policy * Update IAM permissions for reading track ingest * Enable full temporal search on CMR granules * Add capability to download shapefile granule to count features * Update granule UR to include .zip * Count features via Hydrocron table query * Remove unnecessary s3 permissions * Remove whitespace from blank line * Update cryptography to 43.0.1 * Update track ingest table operations * Update changelog with issue * update dependencies * upgrade geopandas * update dependencies --------- Co-authored-by: nikki-t <nikki-t@users.noreply.github.com> Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> Co-authored-by: frankinspace <frankinspace@users.noreply.github.com> Co-authored-by: Victoria McDonald <49625194+torimcd@users.noreply.github.com> Co-authored-by: Cassie Nickles <44206543+cassienickles@users.noreply.github.com> Co-authored-by: cassienickles <cassienickles@users.noreply.github.com> Co-authored-by: podaac-cicd[bot] <podaac-cicd[bot]@users.noreply.github.com> Co-authored-by: Victoria McDonald <victoria.mcdonald@jpl.nasa.gov> Co-authored-by: torimcd <torimcd@users.noreply.github.com>

* /version 1.3.0a0 * Update build.yml * /version 1.3.0a1 * /version 1.3.0a2 * Feature/issue 175 - Update docs to point to OPS (#176) * changelog * update examples, remove load_data readme, info moved to wiki * Dependency update to fix snyk scan * issues/101: Support for HTTP Accept header (#172) * Reorganize timeseries code to prep for Accept header * Enable Accept header to return response of specific content-type * Fix whitespace and string continuation * Make error handling consistent and add an additional test where a reach can't be found * Update changelog with issue for unreleased version * Add 415 status code to API definition * Few minor cleanup items * Few minor cleanup items * Update to aiohttp@3.9.4 * Fix dependencies --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.3.0a3 * issues/102: Support compression of API response (#173) * Enable payload compression * Update changelog with issue --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.3.0a4 * Feature/issue 100 Add option to 'compact' GeoJSON result into single feature (#177) * Reorganize timeseries code to prep for Accept header * Enable Accept header to return response of specific content-type * Fix whitespace and string continuation * Make error handling consistent and add an additional test where a reach can't be found * Update changelog with issue for unreleased version * Add 415 status code to API definition * Few minor cleanup items * Few minor cleanup items * Update to aiohttp@3.9.4 * Fix dependencies * Update required query parameters based on current API functionality * Enable return of 'compact' GeoJSON response * Fix linting and add test data * Update documentation for API accept headers and compact GeoJSON response * Fix references to incorrect Accept header examples --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.3.0a5 * Feature/issue 183 (#185) * Provide introduction to timeseries endpoint * Remove _units in fields list * Fix typo * Update examples with Accept headers and compact query parameter * Add issue to changelog * Fix typo in timeseries documentation * Update pymysql * Update pymysql * Provide clarity on accept headers and request parameter fields * /version 1.3.0a6 * Feature/issue 186 Implement API keys (#188) * API Gateway Lambda authorizer to facilitate API keys and usage plans * Unit tests to test Lambda authorizer * Fix terraform file formatting * API Gateway Lambda Authorizer - Lambda function - API Keys and Authorizer definition in OpenAPI spec - API gateway API keys - API gateway usage plans - SSM parameters for API keys * Fix trailing whitespace * Set default region environment variable * Fix SNYK vulnerabilities * Add issue to changelog * Implement custom trusted partner header x-hydrocron-key * Update cryptography for SNYK vulnerability * Update documentation to include API key usage * Update quota and throttle settings for API Gateway * Update API keys documentation to indicate to be implemented * Move API key lookup to Lambda INIT * Remove API key authentication and update API key to x-hydrocron-key * /version 1.3.0a7 * Update changelog for 1.3.0 release * /version 1.4.0a0 * Feature/issue 198 (#207) * Update pylint to deal with errors and fix collection reference * Initial CMR and Hydrocron queries - Includes placeholders for other operations needed to track granule ingest. - GranuleUR query for Hydrocron tables. * Add and set up vcrpy for testing CMR API query * Test track ingest operations - Test CMR and hydrocron queries - Test granuleUR query - Update database to include granuleUR GSI * Update to use track_ingest naming consistently * Initial Lambda function and IAM role definition * Replace deprecated path function with as_file * Add SSM read IAM permissions * Add DynamoDB read permissions * Update track ingest lambda memory * Remove duplicate IAM permissions * Add in permissions to query index * Update changelog * Update changelog description * Use python_cmr for CMR API queries * /version 1.4.0a1 * Add doi to documentation pages (#216) * Update intro.md with DOI * Update overview.md with DOI * /version 1.4.0a2 * issue-193: Add Dynamo DB Table for SWOT Prior Lakes (#209) * add code to handle prior lakes shapefiles, add test prior lake data * update terraform to add prior lake table * fix tests, change to smaller test data file, changelog * linting * reconfigure main load_data method to make more readable and pass linting * lint * lint * fix string casting to lower storage req & update test responses to handle different rounding pattern in coords * update load benchmarking function for linting and add unit test * try parent collection for lakes * update version parsing for parent collection * fix case error * fix lake id reference * add logging to troubleshoot too large features * add item size logging and remove error raise for batch write * clean up logging statements & move numeric_columns assignment * update batch logging statement * Rename constant * Fix temp dir security risk https://rules.sonarsource.com/python/RSPEC-5443/ * Fix temp dir security risk https://rules.sonarsource.com/python/RSPEC-5443/ * fix code coverage calculation --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.4.0a3 * Feature/issue 201 Create a table for tracking granule ingest status (#214) * Define track ingest database and IAM permissions * Update changelog with issue * Modify table structure to support sparse status index * Updated to only apply PITR in ops --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.4.0a4 * Feature/issue 210 - Load large geometry polygons (#219) * add functions to handle null geometries and convert polygons to points * update doi in docs * fix fill null geometries * fix tests and update changelog * /version 1.4.0a5 * Feature/issue 222 - Add granule info to track ingest table on load (#223) * adjust lambdas to populate track ingest table on granule load * changelog * remove test cnm * lint * change error caught when handling checksum * update lambda role permissions to write to track ingest table * fix typo on lake table terraform * set default fill values for checksum and rev date in track status * fix checksum handling in bulk load data * lint * add logging to debug * /version 1.4.0a6 * Add SSM parameter read for last run time * Feature/issue-225: Create one track ingest table per feature type (#226) * add track ingest tables for each feature type and adjust load data to populate * changelog * /version 1.4.0a7 * Feature/issue 196 Add new feature type to query the API for lake data (#224) * Initial API queries for lake data * Unit tests for lake data * Updates after center point calculations - Removed temp code to calculate a point in API - Implemented unit test to test lake data retrieval - Updated fixtures to load in lake data for testing * Add read lake table permissions to lambda timeseries and track ingest roles * Update documenation to include lake data * Updated documentation to include info on lake centerpoints --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.4.0a8 * Feature/issue 205 - Add Confluence API key (#221) * Fix possible variable references before value is assigned * Define Confluence API key and trusted partner plan limits * Define a list of trusted partner keys and store under single parameter * Define API keys as encrypted envrionment variables for Lambda authorizer * Update authorizer and connection class to use KMS to retrieve API keys * Hack to force lambda deployment when ssm value changes (#218) * Add replace_triggered_by to hydrocron_lambda_authorizer * Introduce environment variable that contains random id which will change whenever an API key value changes. This will force lambda to publish new version of the function. * Remove unnecessary hash function * Update to SSM parameter API key storage and null_resource enviroment variable * Update Terraform and AWS provider * Update API key documentation * Set source_code_hash to force deployment of new image * Downgrade AWS provider to 4.0 to remove inline policy errors * Update docs/timeseries.md --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.4.0a9 * /version 1.4.0a10 * changelog for 1.4.0 release * /version 1.5.0a0 * Initial track ingest table query * Fix linting and code style * Implement feature count operations * Enable S3 permissions and set environment variable for track lambda * Fix trailing white spaces and code format * Update docstrings for class methods * Implement run time storage in SSM * Query track table unit tests * Update CHANGELOG with issue * Update SSM run time parameter * Fix trailing whitespace * Fix reference to IAM policy * Enable specification of temporal range to search revision date by * Fix SSM put parameter policy * Update IAM permissions for reading track ingest * Enable full temporal search on CMR granules * Add capability to download shapefile granule to count features * Update granule UR to include .zip * Count features via Hydrocron table query * Remove unnecessary s3 permissions * Remove whitespace from blank line * Update cryptography to 43.0.1 * Update track ingest table operations * Update changelog with issue * update dependencies * upgrade geopandas * update dependencies * Implement operations to publish CNM messages for granules requiring ingest * Implement unit test of publication operations * Fix linting * Add issue to changelog and fix linting * Add EventBridge schedules with appropriate Lambda permissions * Set initial schedule expressions and fix assume policy * Disable eventbridge schedules by default * Update schedule to run weekly * Define 1 hour latency to search by revision_date in CMR --------- Co-authored-by: nikki-t <nikki-t@users.noreply.github.com> Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> Co-authored-by: frankinspace <frankinspace@users.noreply.github.com> Co-authored-by: Victoria McDonald <49625194+torimcd@users.noreply.github.com> Co-authored-by: Cassie Nickles <44206543+cassienickles@users.noreply.github.com> Co-authored-by: cassienickles <cassienickles@users.noreply.github.com> Co-authored-by: podaac-cicd[bot] <podaac-cicd[bot]@users.noreply.github.com> Co-authored-by: Victoria McDonald <victoria.mcdonald@jpl.nasa.gov> Co-authored-by: torimcd <torimcd@users.noreply.github.com>

* /version 1.3.0a0 * Update build.yml * /version 1.3.0a1 * /version 1.3.0a2 * Feature/issue 175 - Update docs to point to OPS (#176) * changelog * update examples, remove load_data readme, info moved to wiki * Dependency update to fix snyk scan * issues/101: Support for HTTP Accept header (#172) * Reorganize timeseries code to prep for Accept header * Enable Accept header to return response of specific content-type * Fix whitespace and string continuation * Make error handling consistent and add an additional test where a reach can't be found * Update changelog with issue for unreleased version * Add 415 status code to API definition * Few minor cleanup items * Few minor cleanup items * Update to aiohttp@3.9.4 * Fix dependencies --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.3.0a3 * issues/102: Support compression of API response (#173) * Enable payload compression * Update changelog with issue --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.3.0a4 * Feature/issue 100 Add option to 'compact' GeoJSON result into single feature (#177) * Reorganize timeseries code to prep for Accept header * Enable Accept header to return response of specific content-type * Fix whitespace and string continuation * Make error handling consistent and add an additional test where a reach can't be found * Update changelog with issue for unreleased version * Add 415 status code to API definition * Few minor cleanup items * Few minor cleanup items * Update to aiohttp@3.9.4 * Fix dependencies * Update required query parameters based on current API functionality * Enable return of 'compact' GeoJSON response * Fix linting and add test data * Update documentation for API accept headers and compact GeoJSON response * Fix references to incorrect Accept header examples --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.3.0a5 * Feature/issue 183 (#185) * Provide introduction to timeseries endpoint * Remove _units in fields list * Fix typo * Update examples with Accept headers and compact query parameter * Add issue to changelog * Fix typo in timeseries documentation * Update pymysql * Update pymysql * Provide clarity on accept headers and request parameter fields * /version 1.3.0a6 * Feature/issue 186 Implement API keys (#188) * API Gateway Lambda authorizer to facilitate API keys and usage plans * Unit tests to test Lambda authorizer * Fix terraform file formatting * API Gateway Lambda Authorizer - Lambda function - API Keys and Authorizer definition in OpenAPI spec - API gateway API keys - API gateway usage plans - SSM parameters for API keys * Fix trailing whitespace * Set default region environment variable * Fix SNYK vulnerabilities * Add issue to changelog * Implement custom trusted partner header x-hydrocron-key * Update cryptography for SNYK vulnerability * Update documentation to include API key usage * Update quota and throttle settings for API Gateway * Update API keys documentation to indicate to be implemented * Move API key lookup to Lambda INIT * Remove API key authentication and update API key to x-hydrocron-key * /version 1.3.0a7 * Update changelog for 1.3.0 release * /version 1.4.0a0 * Feature/issue 198 (#207) * Update pylint to deal with errors and fix collection reference * Initial CMR and Hydrocron queries - Includes placeholders for other operations needed to track granule ingest. - GranuleUR query for Hydrocron tables. * Add and set up vcrpy for testing CMR API query * Test track ingest operations - Test CMR and hydrocron queries - Test granuleUR query - Update database to include granuleUR GSI * Update to use track_ingest naming consistently * Initial Lambda function and IAM role definition * Replace deprecated path function with as_file * Add SSM read IAM permissions * Add DynamoDB read permissions * Update track ingest lambda memory * Remove duplicate IAM permissions * Add in permissions to query index * Update changelog * Update changelog description * Use python_cmr for CMR API queries * /version 1.4.0a1 * Add doi to documentation pages (#216) * Update intro.md with DOI * Update overview.md with DOI * /version 1.4.0a2 * issue-193: Add Dynamo DB Table for SWOT Prior Lakes (#209) * add code to handle prior lakes shapefiles, add test prior lake data * update terraform to add prior lake table * fix tests, change to smaller test data file, changelog * linting * reconfigure main load_data method to make more readable and pass linting * lint * lint * fix string casting to lower storage req & update test responses to handle different rounding pattern in coords * update load benchmarking function for linting and add unit test * try parent collection for lakes * update version parsing for parent collection * fix case error * fix lake id reference * add logging to troubleshoot too large features * add item size logging and remove error raise for batch write * clean up logging statements & move numeric_columns assignment * update batch logging statement * Rename constant * Fix temp dir security risk https://rules.sonarsource.com/python/RSPEC-5443/ * Fix temp dir security risk https://rules.sonarsource.com/python/RSPEC-5443/ * fix code coverage calculation --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.4.0a3 * Feature/issue 201 Create a table for tracking granule ingest status (#214) * Define track ingest database and IAM permissions * Update changelog with issue * Modify table structure to support sparse status index * Updated to only apply PITR in ops --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.4.0a4 * Feature/issue 210 - Load large geometry polygons (#219) * add functions to handle null geometries and convert polygons to points * update doi in docs * fix fill null geometries * fix tests and update changelog * /version 1.4.0a5 * Feature/issue 222 - Add granule info to track ingest table on load (#223) * adjust lambdas to populate track ingest table on granule load * changelog * remove test cnm * lint * change error caught when handling checksum * update lambda role permissions to write to track ingest table * fix typo on lake table terraform * set default fill values for checksum and rev date in track status * fix checksum handling in bulk load data * lint * add logging to debug * /version 1.4.0a6 * Add SSM parameter read for last run time * Feature/issue-225: Create one track ingest table per feature type (#226) * add track ingest tables for each feature type and adjust load data to populate * changelog * /version 1.4.0a7 * Feature/issue 196 Add new feature type to query the API for lake data (#224) * Initial API queries for lake data * Unit tests for lake data * Updates after center point calculations - Removed temp code to calculate a point in API - Implemented unit test to test lake data retrieval - Updated fixtures to load in lake data for testing * Add read lake table permissions to lambda timeseries and track ingest roles * Update documenation to include lake data * Updated documentation to include info on lake centerpoints --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.4.0a8 * Feature/issue 205 - Add Confluence API key (#221) * Fix possible variable references before value is assigned * Define Confluence API key and trusted partner plan limits * Define a list of trusted partner keys and store under single parameter * Define API keys as encrypted envrionment variables for Lambda authorizer * Update authorizer and connection class to use KMS to retrieve API keys * Hack to force lambda deployment when ssm value changes (#218) * Add replace_triggered_by to hydrocron_lambda_authorizer * Introduce environment variable that contains random id which will change whenever an API key value changes. This will force lambda to publish new version of the function. * Remove unnecessary hash function * Update to SSM parameter API key storage and null_resource enviroment variable * Update Terraform and AWS provider * Update API key documentation * Set source_code_hash to force deployment of new image * Downgrade AWS provider to 4.0 to remove inline policy errors * Update docs/timeseries.md --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.4.0a9 * /version 1.4.0a10 * changelog for 1.4.0 release * update dependencies for 1.4.0 release * /version 1.5.0a0 * fix CMR query in UAT * /version 1.4.0rc1 * fix typo in load_data lambda * /version 1.4.0rc2 * Initial track ingest table query * Fix linting and code style * Implement feature count operations * Enable S3 permissions and set environment variable for track lambda * Fix trailing white spaces and code format * Update docstrings for class methods * Implement run time storage in SSM * Query track table unit tests * Update CHANGELOG with issue * Update SSM run time parameter * Fix trailing whitespace * Fix reference to IAM policy * Enable specification of temporal range to search revision date by * Fix SSM put parameter policy * Update IAM permissions for reading track ingest * Enable full temporal search on CMR granules * Add capability to download shapefile granule to count features * Update granule UR to include .zip * Count features via Hydrocron table query * Remove unnecessary s3 permissions * Remove whitespace from blank line * Update cryptography to 43.0.1 * Update track ingest table operations * Update changelog with issue * update dependencies * upgrade geopandas * update dependencies * fix index on rev date in load data lambda * update dependencies * lint readme * /version 1.4.0rc3 * /version 1.4.0rc4 * Implement operations to publish CNM messages for granules requiring ingest * Implement unit test of publication operations * Fix linting * Add issue to changelog and fix linting * Add EventBridge schedules with appropriate Lambda permissions * Set initial schedule expressions and fix assume policy * fix cmr env search by venue * /version 1.4.0rc5 * Disable eventbridge schedules by default * Update schedule to run weekly * Define 1 hour latency to search by revision_date in CMR * Allow CMR UAT query based on HYDROCRON_ENV environment variable * Update unit tests to accomodate UAT CMR query * Add earthdata login credentials to Lambda * Add issue to changelog * Fix linting white space --------- Co-authored-by: nikki-t <nikki-t@users.noreply.github.com> Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> Co-authored-by: frankinspace <frankinspace@users.noreply.github.com> Co-authored-by: Victoria McDonald <49625194+torimcd@users.noreply.github.com> Co-authored-by: Cassie Nickles <44206543+cassienickles@users.noreply.github.com> Co-authored-by: cassienickles <cassienickles@users.noreply.github.com> Co-authored-by: podaac-cicd[bot] <podaac-cicd[bot]@users.noreply.github.com> Co-authored-by: Victoria McDonald <victoria.mcdonald@jpl.nasa.gov> Co-authored-by: torimcd <torimcd@users.noreply.github.com>

nikki-t added 13 commits July 18, 2024 14:16

Update pylint to deal with errors and fix collection reference

672b67d

Initial CMR and Hydrocron queries

816dd44

- Includes placeholders for other operations needed to track granule ingest. - GranuleUR query for Hydrocron tables.

Add and set up vcrpy for testing CMR API query

f785aa2

Test track ingest operations

8641bc1

- Test CMR and hydrocron queries - Test granuleUR query - Update database to include granuleUR GSI

Update to use track_ingest naming consistently

12b3621

Initial Lambda function and IAM role definition

704c389

Replace deprecated path function with as_file

1cd9dfe

Add SSM read IAM permissions

51727b5

Add DynamoDB read permissions

4d5fb5a

Update track ingest lambda memory

492f70c

Remove duplicate IAM permissions

6020608

Add in permissions to query index

e3024fa

Update changelog

d3a8d2d

nikki-t requested review from torimcd and frankinspace July 19, 2024 19:46

nikki-t self-assigned this Jul 19, 2024

nikki-t linked an issue Jul 19, 2024 that may be closed by this pull request

Implement queries for cmr and hydrocron #198

Closed

frankinspace reviewed Jul 19, 2024

View reviewed changes

Update changelog description

266595f

nikki-t temporarily deployed to SIT July 19, 2024 20:52 — with GitHub Actions Inactive

Use python_cmr for CMR API queries

4509043

nikki-t temporarily deployed to SIT July 24, 2024 13:55 — with GitHub Actions Inactive

nikki-t temporarily deployed to SIT July 24, 2024 14:11 — with GitHub Actions Inactive

torimcd approved these changes Jul 30, 2024

View reviewed changes

nikki-t merged commit 99044f1 into develop Jul 31, 2024
9 checks passed

nikki-t deleted the feature/issue-198 branch July 31, 2024 14:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/issue 198 #207

Feature/issue 198 #207

nikki-t commented Jul 19, 2024

frankinspace Jul 19, 2024

torimcd Jul 30, 2024

nikki-t Jul 31, 2024

nikki-t commented Jul 24, 2024

Feature/issue 198 #207

Feature/issue 198 #207

Conversation

nikki-t commented Jul 19, 2024

Description

Overview of work done

Overview of verification done

Overview of integration done

PR checklist:

frankinspace Jul 19, 2024

Choose a reason for hiding this comment

torimcd Jul 30, 2024

Choose a reason for hiding this comment

nikki-t Jul 31, 2024

Choose a reason for hiding this comment

nikki-t commented Jul 24, 2024