Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not fail cleanup-dbt-resources workflow if databases do not exist #105

Conversation

jeancochrane
Copy link
Contributor

@jeancochrane jeancochrane commented Aug 30, 2023

This PR fixes a bug in the cleanup-dbt-resources workflow that causes it to fail in the (expected) case where it attempts to clean up a database that has not been created by CI. We adjust the underlying cleanup_dbt_resources.sh script to continue execution in cases where the aws glue delete-database command returns a nonzero status with EntityNotFoundException in the output.

Closes #104.

Local testing

I tested this PR locally since it would be a bit of a pain to test on CI. I tested two cases:

  1. Sample output in the case of 404s: I ran the script against the dev target without building any models, and confirmed that it completed execution
  2. Sample output in the case of unexpected error: I adjusted the script to add a bogus --asdf flag to the aws glue delete-database call, and confirmed that it stopped execution

Select a dropdown below to see the corresponding test output.

Sample output in the case of 404s
(venv) jecochr@21CCAO-LAPTOP54:~/code/data-architecture/dbt$ ../.github/scripts/cleanup_dbt_resources.sh dev
Deleting the following schemas from Athena:

"dev_jecochr_census"
"dev_jecochr_default"
"dev_jecochr_location"
"dev_jecochr_model"
"dev_jecochr_proximity"
"dev_jecochr_reporting"
"dev_jecochr_rpie"

An error occurred (EntityNotFoundException) when calling the DeleteDatabase operation: Database dev_jecochr_census not found.
Continuing execution due to expected 404 response.

An error occurred (EntityNotFoundException) when calling the DeleteDatabase operation: Database dev_jecochr_default not found.
Continuing execution due to expected 404 response.

An error occurred (EntityNotFoundException) when calling the DeleteDatabase operation: Database dev_jecochr_location not found.
Continuing execution due to expected 404 response.

An error occurred (EntityNotFoundException) when calling the DeleteDatabase operation: Database dev_jecochr_model not found.
Continuing execution due to expected 404 response.

An error occurred (EntityNotFoundException) when calling the DeleteDatabase operation: Database dev_jecochr_proximity not found.
Continuing execution due to expected 404 response.

An error occurred (EntityNotFoundException) when calling the DeleteDatabase operation: Database dev_jecochr_reporting not found.
Continuing execution due to expected 404 response.

An error occurred (EntityNotFoundException) when calling the DeleteDatabase operation: Database dev_jecochr_rpie not found.
Continuing execution due to expected 404 response.
Sample output in the case of unexpected error
(venv) jecochr@21CCAO-LAPTOP54:~/code/data-architecture/dbt$ ../.github/scripts/cleanup_dbt_resources.sh dev
Deleting the following schemas from Athena:

"dev_jecochr_census"
"dev_jecochr_default"
"dev_jecochr_location"
"dev_jecochr_model"
"dev_jecochr_proximity"
"dev_jecochr_reporting"
"dev_jecochr_rpie"

usage: aws [options] <command> <subcommand> [<subcommand> ...] [parameters]
To see help text, you can run:

  aws help
  aws <command> help
  aws <command> <subcommand> help

Unknown options: --asdf
xargs: bash: exited with status 255; aborting

@jeancochrane jeancochrane marked this pull request as ready for review August 30, 2023 16:00
@jeancochrane jeancochrane requested a review from a team as a code owner August 30, 2023 16:00
Copy link
Member

@dfsnow dfsnow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jeancochrane Nice fix! I should've merged this before Billy's PR since it caused exactly the error described here.

@jeancochrane jeancochrane merged commit 41524a3 into master Aug 31, 2023
3 checks passed
@jeancochrane jeancochrane deleted the jeancochrane/104-data-catalog-fix-cleanup-dbt-resources-workflow-so-it-doesnt-error-in-case-of-deferral branch August 31, 2023 15:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Data catalog] Fix cleanup-dbt-resources workflow so it doesn't error in case of deferral
2 participants