snapshotengine: move sleep to reduce volumes #598
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently the artifact job sleeps a user-defined amount of time to reduce frequency of artifact creation. This resulted in PVCs remaining bound to the kubernetes nodes. With the migration to Digital Ocean a new limit was encountered- only 7 volumes allowed per droplet(node) which was resulting in a need for more droplets.
This PR moves moves the sleep out of the artifact creation job causing it to terminate. The maker job detects this, deletes the job and PVCs and sleeps instead freeing up slots.
Additional info- AWS volume-per-instance limit was at minimum 32 so this was not an issue, but is now on Digital Ocean because of the 7 volume limit per droplet. The volume-per-droplet limit is hard platform limit and not changeable by Digital Ocean support.