Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

wandb server start --upgarde hard to migrate database #94

Open
Jeffrey28 opened this issue Oct 24, 2022 · 26 comments
Open

wandb server start --upgarde hard to migrate database #94

Jeffrey28 opened this issue Oct 24, 2022 · 26 comments

Comments

@Jeffrey28
Copy link

Jeffrey28 commented Oct 24, 2022 •
edited
Loading 

docker logs -f wandb-local
*** Running /etc/my_init.d/00_regen_ssh_host_keys.sh...
*** Running /etc/my_init.d/01_enable-services.sh...
*** Copying services to runit
*** Copying jobber template
*** Enabling production mode
*** Running /etc/my_init.d/02_load-settings.sh...
*** Loading settings...
*** Booting runit daemon...
*** Runit started as PID 84
*** Starting wandb servers...
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/0519aff9-4e68-40de-a96d-78fcd159aea0: permission denied
./run: line 1: kill: (241) - No such process
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/0519aff9-4e68-40de-a96d-78fcd159aea0: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/b04bab63-aa51-4d95-b3e5-6671cae44043: permission denied
./run: line 1: kill: (433) - No such process
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/0519aff9-4e68-40de-a96d-78fcd159aea0: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/b04bab63-aa51-4d95-b3e5-6671cae44043: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/f1527e05-8192-4605-a7f2-d1bcecceed41: permission denied
./run: line 1: kill: (607) - No such process
*** Migrating database...
panic: dial tcp 127.0.0.1:3306: connect: connection refused
@Jeffrey28
Copy link
Author

and another problem:
After I tried cat /vol/env/users.htpasswd in the docker exec -it wandb-local bash
wandb_local is not the right password?

@Jeffrey28
Copy link
Author

image

@vanpelt
Copy link
Contributor

vanpelt commented Oct 26, 2022

Hey @Jeffrey28 we just released a new version of the server that should address this problem. Can you try upgrading again?

@Jeffrey28
Copy link
Author

Actually no. The latest two versions both have the same issue.

@vanpelt
Copy link
Contributor

vanpelt commented Oct 26, 2022

@Jeffrey28 I've verified 0.22.0 works fine when upgrading from 0.20.0. There was a regression in 0.21.0. Can you try running:

docker pull wandb/local:latest
docker stop wandb-local
wandb server start

If that doesn't work, can you share the output of docker logs -f wandb-local?

@Jeffrey28
Copy link
Author

Jeffrey28 commented Oct 26, 2022 •
edited
Loading

I‘m in 0.21.0. It works only after I delete the volume. But if I try upgrading to 0.22.0, it failed.

docker logs -f wandb-local
*** Running /etc/my_init.d/00_regen_ssh_host_keys.sh...
*** Running /etc/my_init.d/01_enable-services.sh...
*** Copying services to runit
*** Copying jobber template
*** Enabling production mode
*** Running /etc/my_init.d/02_load-settings.sh...
*** Loading settings...
*** Booting runit daemon...
*** Runit started as PID 40
*** Starting wandb servers...
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/ab31d888-89d6-4260-9dfc-c6a9a779d474: permission denied
./run: line 1: kill: (118) - No such process
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/ab31d888-89d6-4260-9dfc-c6a9a779d474: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/1a5e6dae-51fe-459b-bfff-a0674090640e: permission denied
./run: line 1: kill: (279) - No such process
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/ab31d888-89d6-4260-9dfc-c6a9a779d474: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/1a5e6dae-51fe-459b-bfff-a0674090640e: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/0c8a61a9-9a9f-43ec-89b9-6643dadf6ef4: permission denied
./run: line 1: kill: (392) - No such process
*** Minio still starting up...
*** Migrating database...
panic: dial tcp 127.0.0.1:3306: connect: connection refused

goroutine 1 [running]:
github.com/wandb/core/services/gorilla/cmd.(*migrateCommander).MainCmd(0xc000d143e0, {0xc000c88540, 0x2, 0x2})
	/mnt/ramdisk/core/services/gorilla/cmd/migrate.go:88 +0x9b4
main.main()
	/mnt/ramdisk/core/services/gorilla/cmd/megabinary/main.go:57 +0x227
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/ab31d888-89d6-4260-9dfc-c6a9a779d474: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/1a5e6dae-51fe-459b-bfff-a0674090640e: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/0c8a61a9-9a9f-43ec-89b9-6643dadf6ef4: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/d87586eb-d242-4200-83c7-e222b003e207: permission denied
./run: line 1: kill: (510) - No such process
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/1a5e6dae-51fe-459b-bfff-a0674090640e: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/0c8a61a9-9a9f-43ec-89b9-6643dadf6ef4: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/d87586eb-d242-4200-83c7-e222b003e207: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/4f95c63e-6211-4dd4-b57f-18c2bf5fe5ba: permission denied
./run: line 1: kill: (706) - No such process
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/0c8a61a9-9a9f-43ec-89b9-6643dadf6ef4: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/d87586eb-d242-4200-83c7-e222b003e207: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/4f95c63e-6211-4dd4-b57f-18c2bf5fe5ba: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/56af9bcf-d2ee-4e75-a22e-e6c200234016: permission denied
./run: line 1: kill: (822) - No such process
*** Minio still starting up...
*** Still waiting...
*** Migrating database...
panic: dial tcp 127.0.0.1:3306: connect: connection refused

goroutine 1 [running]:
github.com/wandb/core/services/gorilla/cmd.(*migrateCommander).MainCmd(0xc0001243b0, {0xc00012a2a0, 0x2, 0x2})
	/mnt/ramdisk/core/services/gorilla/cmd/migrate.go:88 +0x9b4
main.main()
	/mnt/ramdisk/core/services/gorilla/cmd/megabinary/main.go:57 +0x227
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/d87586eb-d242-4200-83c7-e222b003e207: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/4f95c63e-6211-4dd4-b57f-18c2bf5fe5ba: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/56af9bcf-d2ee-4e75-a22e-e6c200234016: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/f5cb6796-a0e1-453e-a3b0-23b44e3556b3: permission denied
./run: line 1: kill: (939) - No such process
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/4f95c63e-6211-4dd4-b57f-18c2bf5fe5ba: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/56af9bcf-d2ee-4e75-a22e-e6c200234016: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/f5cb6796-a0e1-453e-a3b0-23b44e3556b3: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/110ebc38-c955-4775-a5a0-e0060247909d: permission denied
./run: line 1: kill: (1091) - No such process
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/56af9bcf-d2ee-4e75-a22e-e6c200234016: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/f5cb6796-a0e1-453e-a3b0-23b44e3556b3: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/110ebc38-c955-4775-a5a0-e0060247909d: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/089850a3-5e56-4e71-96ba-13dfa330fd63: permission denied
./run: line 1: kill: (1207) - No such process
*** Minio still starting up...
*** Still waiting...
*** Migrating database...
panic: dial tcp 127.0.0.1:3306: connect: connection refused

goroutine 1 [running]:
github.com/wandb/core/services/gorilla/cmd.(*migrateCommander).MainCmd(0xc00013c370, {0xc0000e2380, 0x2, 0x2})
	/mnt/ramdisk/core/services/gorilla/cmd/migrate.go:88 +0x9b4
main.main()
	/mnt/ramdisk/core/services/gorilla/cmd/megabinary/main.go:57 +0x227
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/f5cb6796-a0e1-453e-a3b0-23b44e3556b3: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/110ebc38-c955-4775-a5a0-e0060247909d: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/089850a3-5e56-4e71-96ba-13dfa330fd63: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/d11c1bd9-0d43-44f5-8378-c01f708cf70d: permission denied
./run: line 1: kill: (1327) - No such process
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/110ebc38-c955-4775-a5a0-e0060247909d: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/089850a3-5e56-4e71-96ba-13dfa330fd63: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/d11c1bd9-0d43-44f5-8378-c01f708cf70d: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/abf0e4a9-ec36-4d48-8ee1-13e9e5db74f0: permission denied
./run: line 1: kill: (1468) - No such process
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/089850a3-5e56-4e71-96ba-13dfa330fd63: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/d11c1bd9-0d43-44f5-8378-c01f708cf70d: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/abf0e4a9-ec36-4d48-8ee1-13e9e5db74f0: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/5b2e2203-f758-4a9a-a686-ba8806e0fe9c: permission denied
./run: line 1: kill: (1628) - No such process
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/d11c1bd9-0d43-44f5-8378-c01f708cf70d: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/abf0e4a9-ec36-4d48-8ee1-13e9e5db74f0: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/5b2e2203-f758-4a9a-a686-ba8806e0fe9c: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/d452a52b-aac4-4b76-b42a-b64a84405e85: permission denied
./run: line 1: kill: (1742) - No such process
*** Minio still starting up...

@Jeffrey28
Copy link
Author

Recent versions cause me to fear upgrading

@vanpelt
Copy link
Contributor

vanpelt commented Oct 26, 2022 •
edited
Loading

We understand your fear and are adding more tests to prevent upgrade failures like this in the future.

0.21.0 had a regression which caused volumes created previously to fail upgrading. If your volume was created in any version prior to 0.21.0, then upgrading to 0.22.0 would work. If you created a fresh volume with 0.21.0 you'll need to either delete the volume, or run the following after upgrading:

docker exec -it wandb-local sudo fix_permissions.sh /vol
docker restart wandb-local

@exalate-issue-sync
Copy link

WandB Internal User commented:
Jeffrey28 commented:
Actually no. The latest two versions both have the same issue.

@exalate-issue-sync
Copy link

WandB Internal User commented:
Jeffrey28 commented:
I‘m in 0.21.0. It works only after I delete the volume. But if I try upgrading, it failed.

docker logs -f wandb-local
*** Running /etc/my_init.d/00_regen_ssh_host_keys.sh...
*** Running /etc/my_init.d/01_enable-services.sh...
*** Copying services to runit
*** Copying jobber template
*** Enabling production mode
*** Running /etc/my_init.d/02_load-settings.sh...
*** Loading settings...
*** Booting runit daemon...
*** Runit started as PID 40
*** Starting wandb servers...
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/ab31d888-89d6-4260-9dfc-c6a9a779d474: permission denied
./run: line 1: kill: (118) - No such process
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/ab31d888-89d6-4260-9dfc-c6a9a779d474: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/1a5e6dae-51fe-459b-bfff-a0674090640e: permission denied
./run: line 1: kill: (279) - No such process
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/ab31d888-89d6-4260-9dfc-c6a9a779d474: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/1a5e6dae-51fe-459b-bfff-a0674090640e: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/0c8a61a9-9a9f-43ec-89b9-6643dadf6ef4: permission denied
./run: line 1: kill: (392) - No such process
*** Minio still starting up...
*** Migrating database...
panic: dial tcp 127.0.0.1:3306: connect: connection refused

goroutine 1 [running]:
github.com/wandb/core/services/gorilla/cmd.(*migrateCommander).MainCmd(0xc000d143e0, {0xc000c88540, 0x2, 0x2})
	/mnt/ramdisk/core/services/gorilla/cmd/migrate.go:88 +0x9b4
main.main()
	/mnt/ramdisk/core/services/gorilla/cmd/megabinary/main.go:57 +0x227
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/ab31d888-89d6-4260-9dfc-c6a9a779d474: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/1a5e6dae-51fe-459b-bfff-a0674090640e: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/0c8a61a9-9a9f-43ec-89b9-6643dadf6ef4: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/d87586eb-d242-4200-83c7-e222b003e207: permission denied
./run: line 1: kill: (510) - No such process
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/1a5e6dae-51fe-459b-bfff-a0674090640e: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/0c8a61a9-9a9f-43ec-89b9-6643dadf6ef4: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/d87586eb-d242-4200-83c7-e222b003e207: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/4f95c63e-6211-4dd4-b57f-18c2bf5fe5ba: permission denied
./run: line 1: kill: (706) - No such process
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/0c8a61a9-9a9f-43ec-89b9-6643dadf6ef4: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/d87586eb-d242-4200-83c7-e222b003e207: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/4f95c63e-6211-4dd4-b57f-18c2bf5fe5ba: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/56af9bcf-d2ee-4e75-a22e-e6c200234016: permission denied
./run: line 1: kill: (822) - No such process
*** Minio still starting up...
*** Still waiting...
*** Migrating database...
panic: dial tcp 127.0.0.1:3306: connect: connection refused

goroutine 1 [running]:
github.com/wandb/core/services/gorilla/cmd.(*migrateCommander).MainCmd(0xc0001243b0, {0xc00012a2a0, 0x2, 0x2})
	/mnt/ramdisk/core/services/gorilla/cmd/migrate.go:88 +0x9b4
main.main()
	/mnt/ramdisk/core/services/gorilla/cmd/megabinary/main.go:57 +0x227
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/d87586eb-d242-4200-83c7-e222b003e207: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/4f95c63e-6211-4dd4-b57f-18c2bf5fe5ba: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/56af9bcf-d2ee-4e75-a22e-e6c200234016: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/f5cb6796-a0e1-453e-a3b0-23b44e3556b3: permission denied
./run: line 1: kill: (939) - No such process
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/4f95c63e-6211-4dd4-b57f-18c2bf5fe5ba: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/56af9bcf-d2ee-4e75-a22e-e6c200234016: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/f5cb6796-a0e1-453e-a3b0-23b44e3556b3: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/110ebc38-c955-4775-a5a0-e0060247909d: permission denied
./run: line 1: kill: (1091) - No such process
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/56af9bcf-d2ee-4e75-a22e-e6c200234016: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/f5cb6796-a0e1-453e-a3b0-23b44e3556b3: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/110ebc38-c955-4775-a5a0-e0060247909d: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/089850a3-5e56-4e71-96ba-13dfa330fd63: permission denied
./run: line 1: kill: (1207) - No such process
*** Minio still starting up...
*** Still waiting...
*** Migrating database...
panic: dial tcp 127.0.0.1:3306: connect: connection refused

goroutine 1 [running]:
github.com/wandb/core/services/gorilla/cmd.(*migrateCommander).MainCmd(0xc00013c370, {0xc0000e2380, 0x2, 0x2})
	/mnt/ramdisk/core/services/gorilla/cmd/migrate.go:88 +0x9b4
main.main()
	/mnt/ramdisk/core/services/gorilla/cmd/megabinary/main.go:57 +0x227
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/f5cb6796-a0e1-453e-a3b0-23b44e3556b3: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/110ebc38-c955-4775-a5a0-e0060247909d: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/089850a3-5e56-4e71-96ba-13dfa330fd63: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/d11c1bd9-0d43-44f5-8378-c01f708cf70d: permission denied
./run: line 1: kill: (1327) - No such process
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/110ebc38-c955-4775-a5a0-e0060247909d: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/089850a3-5e56-4e71-96ba-13dfa330fd63: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/d11c1bd9-0d43-44f5-8378-c01f708cf70d: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/abf0e4a9-ec36-4d48-8ee1-13e9e5db74f0: permission denied
./run: line 1: kill: (1468) - No such process
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/089850a3-5e56-4e71-96ba-13dfa330fd63: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/d11c1bd9-0d43-44f5-8378-c01f708cf70d: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/abf0e4a9-ec36-4d48-8ee1-13e9e5db74f0: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/5b2e2203-f758-4a9a-a686-ba8806e0fe9c: permission denied
./run: line 1: kill: (1628) - No such process
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/d11c1bd9-0d43-44f5-8378-c01f708cf70d: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/abf0e4a9-ec36-4d48-8ee1-13e9e5db74f0: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/5b2e2203-f758-4a9a-a686-ba8806e0fe9c: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/d452a52b-aac4-4b76-b42a-b64a84405e85: permission denied
./run: line 1: kill: (1742) - No such process
*** Minio still starting up...

@exalate-issue-sync
Copy link

WandB Internal User commented:
Jeffrey28 commented:
Recent versions cause me to fear upgrading

@exalate-issue-sync
Copy link

WandB Internal User commented:
vanpelt commented:
We understand your fear and our adding more tests to prevent upgrade failures like this in the future.

0.21.0 had a regression which caused volumes created previously to fail upgrading. If your volume was created in any version prior to 0.21.0, then upgrading to 0.22.0 would work. If you created a fresh volume with 0.21.0 you'll need to either delete the volume, or run the following after upgrading:

docker exec -it wandb-local sudo fix_permissions.sh /vol
docker restart wandb-local

@Jeffrey28
Copy link
Author

Thanks, it works now with these two lines.

@Jeffrey28
Copy link
Author

But coming with the problem:
image

@Jeffrey28
Copy link
Author

It has to remove the volume. But the data will be missing.

@vanpelt
Copy link
Contributor

vanpelt commented Oct 26, 2022

Sorry about the 2 lines not working, we'll investigate and reproduce the workspace error and provide lines that work. You should never lose data, that's unacceptable.

@vanpelt
Copy link
Contributor

vanpelt commented Oct 26, 2022

@Jeffrey28 I just verified lines that work when upgrading from a volume that was first mounted on version 0.21.0, to a server running 0.22.0:

docker exec -it wandb-local sudo chown -R wandb /vol
wandb server stop
wandb server start

@exalate-issue-sync
Copy link

WandB Internal User commented:
Jeffrey28 commented:

Thanks, it works now with these two lines.

@exalate-issue-sync
Copy link

WandB Internal User commented:
vanpelt commented:
Sorry about the 2 lines not working, we'll investigate and reproduce the workspace error and provide lines that work. You should never lose data, that's unacceptable.

@exalate-issue-sync
Copy link

Luis Bergua commented:
Hi Jeffrey, I just wanted to ask you if these lines worked for you when upgrading the 0.21.0 volume to 0.22.0

docker exec -it wandb-local sudo chown -R wandb /volwandb server stopwandb server start

Thanks!

@Jeffrey28
Copy link
Author

Luis Bergua commented: Hi Jeffrey, I just wanted to ask you if these lines worked for you when upgrading the 0.21.0 volume to 0.22.0

docker exec -it wandb-local sudo chown -R wandb /volwandb server stopwandb server start

Thanks!

It works but the data has some problems. Therefore, I have to reset the database.

@luisbergua
Copy link

Thanks for your answer @Jeffrey28, would you mind explaining me what kind of problems has the data and so I can try to help you or have you reset the database and it has solved these problems?

@Jeffrey28
Copy link
Author

After reseting the database, it works well now.

@luisbergua
Copy link

Great to hear it @Jeffrey28! I'll then close this issue as it seems to be solved. Thanks!

@exalate-issue-sync exalate-issue-sync bot reopened this Nov 14, 2022
@exalate-issue-sync
Copy link

WandB Internal User commented:
Jeffrey28 commented:
After reseting the database, it works well now.

@exalate-issue-sync
Copy link

WandB Internal User commented:
luisbergua commented:
Great to hear it @Jeffrey28! I'll then close this issue as it seems to be solved. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@vanpelt @Jeffrey28 @luisbergua