Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Complete an integration test in the SIT environment #238

Open
nikki-t opened this issue Sep 24, 2024 · 6 comments
Open

Complete an integration test in the SIT environment #238

nikki-t opened this issue Sep 24, 2024 · 6 comments
Assignees
@nikki-t
Labels
enhancement New feature or request
Milestone
Track status of g...

Comments

@nikki-t
Copy link
Collaborator

nikki-t commented Sep 24, 2024

To determine if the track ingest operations function to reconcile any granules that were not ingested into Hydrocron, a test should be performed in the services SIT environment. The test steps for SIT are documented here: https://wiki.jpl.nasa.gov/display/PD/Hydrocron#Hydrocron-SITtest(1.5.0)
@nikki-t nikki-t self-assigned this Sep 24, 2024
@nikki-t nikki-t added the enhancement New feature or request label Sep 24, 2024
@nikki-t
Copy link
Collaborator Author

nikki-t commented Sep 24, 2024 •
edited
Loading

Integration Test (Reaches)

Test 1) Run track ingest Lambda on temporal range for reaches: 2024-09-13T01:00:00 to 2024-09-13T04:00:00.

5 granules NOT in Hydrocron:

  • SWOT_L2_HR_RiverSP_Reach_021_074_SA_20240913T033713_20240913T033722_PIC0_01.zip
  • SWOT_L2_HR_RiverSP_Reach_021_074_NA_20240913T032024_20240913T032032_PIC0_01.zip
  • SWOT_L2_HR_RiverSP_Reach_021_074_AR_20240913T031404_20240913T031411_PIC0_01.zip
  • SWOT_L2_HR_RiverSP_Reach_021_072_GR_20240913T013154_20240913T013158_PIC0_01.zip
  • SWOT_L2_HR_RiverSP_Reach_021_071_AR_20240913T012106_20240913T012111_PIC0_01.zip

Operations:

a. Wipe out track ingest tables.
b. Remove granules from SWOT tables.
c. Execute track ingest manually.
d. Track ingest will query CMR and pull in 5 granules that are not in Hydrocron.
e. Track ingest will send a CNM message to the Hydrocron CNM Lambda to ingest 5 granules.
f. Track ingest will update its table with the 5 granules set to a status of "to_ingest".

Track Ingest Logs

2024-09-25T13:16:33.598Z [INFO] 2024-09-25T13:16:33.598Z Located 5 granules in CMR.
2024-09-25T13:16:33.786Z [INFO] 2024-09-25T13:16:33.786Z Located 5 granules NOT in Hydrocron.
2024-09-25T13:16:33.809Z [INFO] 2024-09-25T13:16:33.809Z Located 0 granules with 'to_ingest' status.
2024-09-25T13:16:33.809Z [INFO] 2024-09-25T13:16:33.809Z Located 5 granules that require ingestion.
2024-09-25T13:16:33.809Z [INFO] 2024-09-25T13:16:33.809Z Located 0 granules that are already ingested.
...
2024-09-25T13:16:37.972Z [INFO] 2024-09-25T13:16:37.972Z SWOT_L2_HR_RiverSP_Reach_021_071_AR_20240913T012106_20240913T012111_PIC0_01 message published to SNS Topic: svc-hydrocron-sit-cnm-response
2024-09-25T13:16:37.987Z [INFO] 2024-09-25T13:16:37.987Z SWOT_L2_HR_RiverSP_Reach_021_072_GR_20240913T013154_20240913T013158_PIC0_01 message published to SNS Topic: svc-hydrocron-sit-cnm-response	
2024-09-25T13:16:38.003Z [INFO] 2024-09-25T13:16:38.002Z SWOT_L2_HR_RiverSP_Reach_021_074_AR_20240913T031404_20240913T031411_PIC0_01 message published to SNS Topic: svc-hydrocron-sit-cnm-response	
2024-09-25T13:16:38.017Z [INFO] 2024-09-25T13:16:38.017Z SWOT_L2_HR_RiverSP_Reach_021_074_NA_20240913T032024_20240913T032032_PIC0_01 message published to SNS Topic: svc-hydrocron-sit-cnm-response	
2024-09-25T13:16:38.032Z [INFO] 2024-09-25T13:16:38.032Z SWOT_L2_HR_RiverSP_Reach_021_074_SA_20240913T033713_20240913T033722_PIC0_01 message published to SNS Topic: svc-hydrocron-sit-cnm-response

Confirmed CNM Lambda was invoked 5 times (one for each granule).
Confirmed Load Granule Lambda was invoked 5 times (one for each granule).

Track ingest table was updated:
test1

Test 2) Run track ingest Lambda on temporal range for reaches: 2024-09-13T10:00:00 to 2024-09-13T12:00:00.

3 granules NOT in Hydrocron:

  • SWOT_L2_HR_RiverSP_Reach_021_084_SI_20240913T114534_20240913T114538_PIC0_01.zip
  • SWOT_L2_HR_RiverSP_Reach_021_082_AR_20240913T100914_20240913T100916_PIC0_01.zip
  • SWOT_L2_HR_RiverSP_Reach_021_082_SI_20240913T100755_20240913T100756_PIC0_01.zip

Operations:

b. Track ingest will query CMR and pull in 3 granules that are not in Hydrocron.
c. Track ingest will send a CNM message to the Hydrocron CNM Lambda to ingest 3 granules.
d. Track ingest will update its table with the 3 granules set to a status of "to_ingest".
e. Track ingest will update its table with 5 granules from previous test 1) to remove "to_ingest" status and set actual feature count (should be equal to expected feature count).

Track Ingest Logs

2024-09-25T13:23:39.090Z [INFO] 2024-09-25T13:23:39.090Z Located 3 granules in CMR.
2024-09-25T13:23:39.224Z [INFO] 2024-09-25T13:23:39.224Z Located 3 granules NOT in Hydrocron.
2024-09-25T13:23:39.233Z [INFO] 2024-09-25T13:23:39.233Z Located 5 granules with 'to_ingest' status.
2024-09-25T13:23:39.412Z [INFO] 2024-09-25T13:23:39.411Z Located 3 granules that require ingestion.
2024-09-25T13:23:39.412Z [INFO] 2024-09-25T13:23:39.412Z Located 5 granules that are already ingested. 
...
2024-09-25T13:23:41.507Z [INFO] 2024-09-25T13:23:41.507Z SWOT_L2_HR_RiverSP_Reach_021_082_SI_20240913T100755_20240913T100756_PIC0_01 message published to SNS Topic: svc-hydrocron-sit-cnm-response
2024-09-25T13:23:41.521Z [INFO] 2024-09-25T13:23:41.521Z SWOT_L2_HR_RiverSP_Reach_021_082_AR_20240913T100914_20240913T100916_PIC0_01 message published to SNS Topic: svc-hydrocron-sit-cnm-response
2024-09-25T13:23:41.536Z [INFO] 2024-09-25T13:23:41.536Z SWOT_L2_HR_RiverSP_Reach_021_084_SI_20240913T114534_20240913T114538_PIC0_01 message published to SNS Topic: svc-hydrocron-sit-cnm-response

Confirmed CNM Lambda was invoked 3 times (one for each granule).
Confirmed Load Granule Lambda was invoked 3 times (one for each granule).

Track ingest table was updated:
test2

Test 3) Run track ingest Lambda on temporal range for reaches: 2024-09-13T13:00:00 to 2024-09-13T23:59:59.

6 granules NOT in Hydrocron:

  • SWOT_L2_HR_RiverSP_Reach_021_097_AR_20240913T233753_20240913T233803_PIC0_01.zip
  • SWOT_L2_HR_RiverSP_Reach_021_095_GR_20240913T220139_20240913T220150_PIC0_01.zip
  • SWOT_L2_HR_RiverSP_Reach_021_095_AR_20240913T215308_20240913T215319_PIC0_01.zip
  • SWOT_L2_HR_RiverSP_Reach_021_095_NA_20240913T215139_20240913T215139_PIC0_01.zip
  • SWOT_L2_HR_RiverSP_Reach_021_086_AU_20240913T140657_20240913T140704_PIC0_01.zip
  • SWOT_L2_HR_RiverSP_Reach_021_086_SI_20240913T132957_20240913T133003_PIC0_01.zip

Operations:

b. Track ingest will query CMR and pull in 6 granules that are not in Hydrocron.
c. Track ingest will send a CNM message to the Hydrocron CNM Lambda to ingest 6 granules.
d. Track ingest will update its table with the 6 granules set to a status of "to_ingest".
e. Track ingest will update its table with 3 granules from previous test 2) to remove "to_ingest" status and set actual feature count (should be equal to expected feature count).

Track Ingest Logs

2024-09-25T13:30:29.288Z [INFO] 2024-09-25T13:30:29.288Z Located 6 granules in CMR.	
2024-09-25T13:30:29.534Z [INFO] 2024-09-25T13:30:29.533Z Located 6 granules NOT in Hydrocron.	
2024-09-25T13:30:29.544Z [INFO] 2024-09-25T13:30:29.543Z Located 3 granules with 'to_ingest' status.
2024-09-25T13:30:29.647Z [INFO] 2024-09-25T13:30:29.647Z Located 6 granules that require ingestion.
2024-09-25T13:30:29.647Z [INFO] 2024-09-25T13:30:29.647Z Located 3 granules that are already ingested.
...
2024-09-25T13:30:33.118Z [INFO] 2024-09-25T13:30:33.118Z SWOT_L2_HR_RiverSP_Reach_021_086_SI_20240913T132957_20240913T133003_PIC0_01 message published to SNS Topic: svc-hydrocron-sit-cnm-response
2024-09-25T13:30:33.135Z [INFO] 2024-09-25T13:30:33.132Z SWOT_L2_HR_RiverSP_Reach_021_086_AU_20240913T140657_20240913T140704_PIC0_01 message published to SNS Topic: svc-hydrocron-sit-cnm-response
2024-09-25T13:30:33.148Z [INFO] 2024-09-25T13:30:33.147Z SWOT_L2_HR_RiverSP_Reach_021_095_NA_20240913T215139_20240913T215139_PIC0_01 message published to SNS Topic: svc-hydrocron-sit-cnm-response
2024-09-25T13:30:33.164Z [INFO] 2024-09-25T13:30:33.164Z SWOT_L2_HR_RiverSP_Reach_021_095_AR_20240913T215308_20240913T215319_PIC0_01 message published to SNS Topic: svc-hydrocron-sit-cnm-response
2024-09-25T13:30:33.177Z [INFO] 2024-09-25T13:30:33.177Z SWOT_L2_HR_RiverSP_Reach_021_095_GR_20240913T220139_20240913T220150_PIC0_01 message published to SNS Topic: svc-hydrocron-sit-cnm-response
2024-09-25T13:30:33.190Z [INFO] 2024-09-25T13:30:33.190Z SWOT_L2_HR_RiverSP_Reach_021_097_AR_20240913T233753_20240913T233803_PIC0_01 message published to SNS Topic: svc-hydrocron-sit-cnm-response

Confirmed CNM Lambda was invoked 6 times (one for each granule).
Confirmed Load Granule Lambda was invoked 6 times (one for each granule).

Track ingest table was updated:
test3

Test 4) Run track ingest Lambda on temporal range for reaches: 2024-09-14T02:00:00 to 2024-09-14T04:00:00.

No new SWOT granules.

Operations:

a. Track ingest will query CMR and pull in 0 granules that are not in Hydrocron.
b. Track ingest will update its table with 6 granules from previous test 3) to remove "to_ingest" status and set actual feature count (should be equal to expected feature count).

Track Ingest Logs

2024-09-25T13:37:32.493Z [INFO] 2024-09-25T13:37:32.493Z Located 0 granules in CMR.
2024-09-25T13:37:32.494Z [INFO] 2024-09-25T13:37:32.493Z Located 0 granules NOT in Hydrocron.
2024-09-25T13:37:32.561Z [INFO] 2024-09-25T13:37:32.561Z Located 6 granules with 'to_ingest' status.
2024-09-25T13:37:32.787Z [INFO] 2024-09-25T13:37:32.787Z Located 0 granules that require ingestion.
2024-09-25T13:37:32.799Z [INFO] 2024-09-25T13:37:32.799Z Located 6 granules that are already ingested.
...
2024-09-25T13:37:33.412Z [INFO] 2024-09-25T13:37:33.412Z Updated hydrocron-swot-reach-track-ingest-table with 6 items.

Confirmed CNM Lambda was invoked 0 times (one for each granule).
Confirmed Load Granule Lambda was invoked 0 times (one for each granule).

Track ingest table was updated:
test4

@nikki-t
Copy link
Collaborator Author

nikki-t commented Sep 25, 2024

Integration Test (Prior Lakes)

Tested in same manner as reaches on the following temporal ranges:

2024-08-25T01:00:00 to 2024-08-25T02:00:00:

  • SWOT_L2_HR_LakeSP_Prior_020_124_AF_20240825T013936_20240825T014737_PIC0_01.zip
  • SWOT_L2_HR_LakeSP_Prior_020_124_EU_20240825T012806_20240825T013248_PIC0_01.zip
  • SWOT_L2_HR_LakeSP_Prior_020_123_AR_20240825T010843_20240825T011234_PIC0_01.zip
  • SWOT_L2_HR_LakeSP_Prior_020_123_NA_20240825T010713_20240825T010904_PIC0_01.zip

2024-08-25T02:00:00 to 2024-08-25T05:00:00:

  • SWOT_L2_HR_LakeSP_Prior_020_127_AR_20240825T043700_20240825T044002_PIC0_01.zip
  • SWOT_L2_HR_LakeSP_Prior_020_127_AU_20240825T041330_20240825T041402_PIC0_01.zip
  • SWOT_L2_HR_LakeSP_Prior_020_126_AF_20240825T031610_20240825T032511_PIC0_01.zip
  • SWOT_L2_HR_LakeSP_Prior_020_126_EU_20240825T030650_20240825T031611_PIC0_01.zip
  • SWOT_L2_HR_LakeSP_Prior_020_125_AR_20240825T025326_20240825T025648_PIC0_01.zip

2024-08-25T05:00:00 to 2024-08-25T07:00:00:

  • SWOT_L2_HR_LakeSP_Prior_020_130_SA_20240825T065156_20240825T065938_PIC0_01.zip
  • SWOT_L2_HR_LakeSP_Prior_020_130_NA_20240825T063507_20240825T063759_PIC0_01.zip
  • SWOT_L2_HR_LakeSP_Prior_020_129_AU_20240825T055143_20240825T060025_PIC0_01.zip

Operations proceeded as expected.

@nikki-t
Copy link
Collaborator Author

nikki-t commented Sep 25, 2024

Integration Test (Nodes)

Tested in same manner as reaches on the following temporal ranges:

2024-09-13T03:00:00 to 2024-09-13T09:00:00:

  • SWOT_L2_HR_RiverSP_Node_021_080_AR_20240913T082347_20240913T082352_PIC0_01.zip
  • SWOT_L2_HR_RiverSP_Node_021_074_SA_20240913T033713_20240913T033722_PIC0_01.zip
  • SWOT_L2_HR_RiverSP_Node_021_074_NA_20240913T032024_20240913T032032_PIC0_01.zip
  • SWOT_L2_HR_RiverSP_Node_021_074_AR_20240913T031404_20240913T031411_PIC0_01.zip

2024-09-13T10:00:00 to 2024-09-13T12:00:00:

  • SWOT_L2_HR_RiverSP_Node_021_084_SI_20240913T114534_20240913T114538_PIC0_01.zip
  • SWOT_L2_HR_RiverSP_Node_021_082_AR_20240913T100914_20240913T100916_PIC0_01.zip
  • SWOT_L2_HR_RiverSP_Node_021_082_SI_20240913T100755_20240913T100756_PIC0_01.zip

2024-09-13T13:00:00 to 2024-09-13T23:59:59:

  • SWOT_L2_HR_RiverSP_Node_021_097_AR_20240913T233753_20240913T233803_PIC0_01.zip
  • SWOT_L2_HR_RiverSP_Node_021_095_GR_20240913T220139_20240913T220150_PIC0_01.zip
  • SWOT_L2_HR_RiverSP_Node_021_095_AR_20240913T215308_20240913T215319_PIC0_01.zip
  • SWOT_L2_HR_RiverSP_Node_021_095_NA_20240913T215139_20240913T215139_PIC0_01.zip
  • SWOT_L2_HR_RiverSP_Node_021_086_AU_20240913T140657_20240913T140704_PIC0_01.zip
  • SWOT_L2_HR_RiverSP_Node_021_086_SI_20240913T132957_20240913T133003_PIC0_01.zip

Operations proceeded as expected.

This was referenced Sep 25, 2024
Merged
Merged
Merged
Merged
@nikki-t
Copy link
Collaborator Author

nikki-t commented Oct 9, 2024

Anomalies from SIT testing:

  • SWOT_L2_HR_RiverSP_Reach granules listed in lambda log output when triggering lambda (and swot-reach granules were also noted in the hydrocron-swot-node-track-ingest-table)
  • expected_feature_count set to -1 instead of an actual number for the Reach granules noted in the hydrocron-swot-node-track-ingest-table
  • When executing the Event JSON the 2nd time per the test, user observed Error in svc-hydrocron-sit-cnm-lambda logs; see /aws/lambda/svc-hydrocron-sit-cnm-lambda 2024/10/09/[$LATEST]ee6639ae9ddf4776839e46c7c15db106

This revealed the need to test for a mismatch between collection shortname and Hydrocron SWOT and track ingest tables: #235 which also includes checking for Observed and Unassigned lakes.

@nikki-t
Copy link
Collaborator Author

nikki-t commented Oct 11, 2024 •
edited
Loading

SIT test also revealed bug where the first query for granules in CMR was set to search in SIT and UAT by parent collection with a bearer token but the second query for granule files was not set up in the same way producing inconsistent results and sometimes not returning associated granule files. This led to the CNM function being invoked with an invalid CNM message. #248

This was referenced Oct 11, 2024
Closed
Merged
@nikki-t
Copy link
Collaborator Author

nikki-t commented Oct 16, 2024 •
edited
Loading

SIT testing has concluded successfully. Results: https://cae-testrail.jpl.nasa.gov/testrail/index.php?/runs/view/48645&group_by=cases:title&group_order=asc

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nikki-t nikki-t

Labels
enhancement New feature or request
Projects
SOTO PI 24.4
Status: 🏗 In progress
Milestone
Track status of granule ingest
Development

No branches or pull requests
1 participant
@nikki-t