Using google cloud commands within Docker when testing on the local provider

[carbocation]

I have a dsub-executed script that uses environment variables to determine which files to localize from storage using gsutil from a large list. (I do this instead of mounting because of the low reliability of gcsfuse.)

When running on the google backend provider, this approach works great even though I haven't explicitly stored any google cloud credentials within the Docker image.

But when running with the local provider, the lack of credentials is a problem when I try to run gsutil within the docker image:

ServiceException: 401 Anonymous caller does not have storage.objects.get access to the Google Cloud Storage object. Permission 'storage.objects.get' denied on resource (or it may not exist).
CommandException: 1 file/object could not be transferred.

Is there a pointer to how we might get gcloud-in-Docker to work for easier local troubleshooting? I tried passing --credentials-file but this didn't seem to have any effect on the local provider.

[mbookman]

Hi @carbocation !

A workaround for this is to make your ~/.config/gcloud directory available inside the container. This can be done with:

  --input-recursive CLOUDSDK_CONFIG_INPUT=${HOME}/.config/gcloud \
  --env CLOUDSDK_CONFIG=/mnt/data/input/file${HOME}/.config/gcloud \

Note that it should be sufficient to use:

  --input-recursive CLOUDSDK_CONFIG=${HOME}/.config/gcloud \

but the local provider sets environment variables too early. It should set the environment variables only for the "user command", but it ends up setting them before the localization command.