Skip to content
This repository has been archived by the owner on Sep 23, 2024. It is now read-only.

Commit

Permalink
Made scripts/deploy_gae.sh deploy version v1. This should reduce the …
Browse files Browse the repository at this point in the history
…time it takes for redeploying.

When a training job fails, give the team back their training minutes.
  • Loading branch information
lizlooney committed Oct 19, 2021
1 parent f419c9f commit fe38ed6
Show file tree
Hide file tree
Showing 2 changed files with 9 additions and 3 deletions.
2 changes: 1 addition & 1 deletion scripts/deploy_gae.sh
Original file line number Diff line number Diff line change
Expand Up @@ -5,5 +5,5 @@ fi


pushd server
gcloud -q app deploy
gcloud -q app deploy --version v1
popd
10 changes: 8 additions & 2 deletions server/storage.py
Original file line number Diff line number Diff line change
Expand Up @@ -1255,8 +1255,14 @@ def model_trainer_started(team_uuid, model_uuid, description, model_folder,
# If the training job has already ended, adjust the team's remaining training time.
if 'train_job_end_time' in model_entity:
team_entity = retrieve_team_entity(team_uuid)
train_job_elapsed_minutes = model_entity['train_job_elapsed_seconds'] / 60
delta = model_entity['max_running_minutes'] - train_job_elapsed_minutes
if model_entity['train_job_state'] == 'FAILED':
# If the job failed, give the max_running_minutes back to the team.
delta = model_entity['max_running_minutes']
else:
# Otherwise, give the different between the max_running_minutes and the actual
# training time back to the team.
train_job_elapsed_minutes = model_entity['train_job_elapsed_seconds'] / 60
delta = model_entity['max_running_minutes'] - train_job_elapsed_minutes
# Don't add the delta if it's negative. The job ran longer than the maximum running
# time that the user specified.
if delta > 0:
Expand Down

0 comments on commit fe38ed6

Please sign in to comment.