Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Infinite loop in stow.Walk #230

Open
inliquid opened this issue Dec 10, 2019 · 3 comments
Open

Infinite loop in stow.Walk #230

inliquid opened this issue Dec 10, 2019 · 3 comments

Comments

@inliquid
Copy link

I'm using stow with Ceph Object Gateway's S3-compatible storage (SWITCH). It aims to be S3-compatible, here is the spec: https://docs.ceph.com/docs/master/radosgw/s3/#features-support

What I got is infinite loop when fetching files (items) located under some test account.

Program (will be stopped forcefully at number of output items == 100):

package main

import (
	"fmt"
	"log"

	"github.com/graymeta/stow"
	"github.com/graymeta/stow/s3"
)

func main() {
	kind := "s3"
	config := stow.ConfigMap{
		s3.ConfigAccessKeyID: "***",
		s3.ConfigSecretKey:   "***",
		s3.ConfigEndpoint: "https://os.zhdk.cloud.switch.ch",
	}
	location, err := stow.Dial(kind, config)
	if err != nil {
		panic(err)
	}
	defer location.Close()

	log.Println("Location:", location)

	containers := []stow.Container{}

	err = stow.WalkContainers(location, stow.NoPrefix, 100, func(c stow.Container, err error) error {
		if err != nil {
			return err
		}
		log.Println("Found container:", c.Name())
		containers = append(containers, c)

		return nil
	})
	if err != nil {
		panic(err)
	}

	i := 0

	for _, c := range containers {
		err = stow.Walk(c, stow.NoPrefix, 10, func(item stow.Item, err error) error {
			if err != nil {
				return err
			}
			lastMod, _ := item.LastMod()
			log.Printf("Found item in '%s': %s: %v\n", c.Name(), item.Name(), lastMod)

			i++
			if i >= 100 {
				return fmt.Errorf("too many items: %d", i)
			}

			return nil
		})
		if err != nil {
			panic(err)
		}
	}
}

Output:

>go run .
2019/12/10 18:47:31 Location: &{map[access_key_id:*** endpoint:https://os.zhdk.cloud.switch.ch secret_key:***] https://os.zhdk.cloud.switch.ch 0xc000006090}
2019/12/10 18:47:32 Found container: atdatasets
2019/12/10 18:47:32 Found container: testupload
2019/12/10 18:47:33 Found item in 'atdatasets': Data/: 2019-08-22 13:36:36 +0000 UTC
2019/12/10 18:47:34 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/: 2019-08-22 13:40:50 +0000 UTC
2019/12/10 18:47:34 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/Ground.tar.gz: 2019-08-22 13:58:36 +0000 UTC
2019/12/10 18:47:34 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/LOGS.tar.gz: 2019-08-22 14:00:34 +0000 UTC
2019/12/10 18:47:35 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/Mutispectral.tar.gz: 2019-08-26 07:46:38 +0000 UTC
2019/12/10 18:47:35 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/Thermal.tar.gz: 2019-08-23 07:21:39 +0000 UTC
2019/12/10 18:47:36 Found item in 'atdatasets': Data/20190821 VyA LAGUNA/: 2019-08-23 10:06:10 +0000 UTC
2019/12/10 18:47:36 Found item in 'atdatasets': Data/20190821 VyA LAGUNA/Ground/: 2019-08-26 10:22:55 +0000 UTC
2019/12/10 18:47:37 Found item in 'atdatasets': Data/20190821 VyA LAGUNA/Ground/DSC01812.JPG: 2019-08-26 10:23:01 +0000 UTC
2019/12/10 18:47:37 Found item in 'atdatasets': Data/20190821 VyA LAGUNA/Ground/DSC01813.JPG: 2019-08-26 10:23:01 +0000 UTC
2019/12/10 18:47:38 Found item in 'atdatasets': Data/: 2019-08-22 13:36:36 +0000 UTC
2019/12/10 18:47:38 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/: 2019-08-22 13:40:50 +0000 UTC
2019/12/10 18:47:39 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/Ground.tar.gz: 2019-08-22 13:58:36 +0000 UTC
2019/12/10 18:47:39 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/LOGS.tar.gz: 2019-08-22 14:00:34 +0000 UTC
2019/12/10 18:47:40 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/Mutispectral.tar.gz: 2019-08-26 07:46:38 +0000 UTC
2019/12/10 18:47:40 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/Thermal.tar.gz: 2019-08-23 07:21:39 +0000 UTC
2019/12/10 18:47:41 Found item in 'atdatasets': Data/20190821 VyA LAGUNA/: 2019-08-23 10:06:10 +0000 UTC
2019/12/10 18:47:41 Found item in 'atdatasets': Data/20190821 VyA LAGUNA/Ground/: 2019-08-26 10:22:55 +0000 UTC
2019/12/10 18:47:42 Found item in 'atdatasets': Data/20190821 VyA LAGUNA/Ground/DSC01812.JPG: 2019-08-26 10:23:01 +0000 UTC
2019/12/10 18:47:42 Found item in 'atdatasets': Data/20190821 VyA LAGUNA/Ground/DSC01813.JPG: 2019-08-26 10:23:01 +0000 UTC
2019/12/10 18:47:43 Found item in 'atdatasets': Data/: 2019-08-22 13:36:36 +0000 UTC
2019/12/10 18:47:44 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/: 2019-08-22 13:40:50 +0000 UTC
2019/12/10 18:47:44 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/Ground.tar.gz: 2019-08-22 13:58:36 +0000 UTC
2019/12/10 18:47:44 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/LOGS.tar.gz: 2019-08-22 14:00:34 +0000 UTC
2019/12/10 18:47:45 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/Mutispectral.tar.gz: 2019-08-26 07:46:38 +0000 UTC
2019/12/10 18:47:45 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/Thermal.tar.gz: 2019-08-23 07:21:39 +0000 UTC
2019/12/10 18:47:46 Found item in 'atdatasets': Data/20190821 VyA LAGUNA/: 2019-08-23 10:06:10 +0000 UTC
2019/12/10 18:47:46 Found item in 'atdatasets': Data/20190821 VyA LAGUNA/Ground/: 2019-08-26 10:22:55 +0000 UTC
2019/12/10 18:47:46 Found item in 'atdatasets': Data/20190821 VyA LAGUNA/Ground/DSC01812.JPG: 2019-08-26 10:23:01 +0000 UTC
2019/12/10 18:47:47 Found item in 'atdatasets': Data/20190821 VyA LAGUNA/Ground/DSC01813.JPG: 2019-08-26 10:23:01 +0000 UTC
2019/12/10 18:47:48 Found item in 'atdatasets': Data/: 2019-08-22 13:36:36 +0000 UTC
2019/12/10 18:47:48 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/: 2019-08-22 13:40:50 +0000 UTC
2019/12/10 18:47:49 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/Ground.tar.gz: 2019-08-22 13:58:36 +0000 UTC
2019/12/10 18:47:49 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/LOGS.tar.gz: 2019-08-22 14:00:34 +0000 UTC
2019/12/10 18:47:50 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/Mutispectral.tar.gz: 2019-08-26 07:46:38 +0000 UTC
2019/12/10 18:47:50 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/Thermal.tar.gz: 2019-08-23 07:21:39 +0000 UTC
2019/12/10 18:47:51 Found item in 'atdatasets': Data/20190821 VyA LAGUNA/: 2019-08-23 10:06:10 +0000 UTC
2019/12/10 18:47:51 Found item in 'atdatasets': Data/20190821 VyA LAGUNA/Ground/: 2019-08-26 10:22:55 +0000 UTC
2019/12/10 18:47:52 Found item in 'atdatasets': Data/20190821 VyA LAGUNA/Ground/DSC01812.JPG: 2019-08-26 10:23:01 +0000 UTC
2019/12/10 18:47:52 Found item in 'atdatasets': Data/20190821 VyA LAGUNA/Ground/DSC01813.JPG: 2019-08-26 10:23:01 +0000 UTC
2019/12/10 18:47:53 Found item in 'atdatasets': Data/: 2019-08-22 13:36:36 +0000 UTC
2019/12/10 18:47:53 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/: 2019-08-22 13:40:50 +0000 UTC
2019/12/10 18:47:54 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/Ground.tar.gz: 2019-08-22 13:58:36 +0000 UTC
2019/12/10 18:47:54 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/LOGS.tar.gz: 2019-08-22 14:00:34 +0000 UTC
2019/12/10 18:47:55 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/Mutispectral.tar.gz: 2019-08-26 07:46:38 +0000 UTC
2019/12/10 18:47:55 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/Thermal.tar.gz: 2019-08-23 07:21:39 +0000 UTC
2019/12/10 18:47:56 Found item in 'atdatasets': Data/20190821 VyA LAGUNA/: 2019-08-23 10:06:10 +0000 UTC
2019/12/10 18:47:56 Found item in 'atdatasets': Data/20190821 VyA LAGUNA/Ground/: 2019-08-26 10:22:55 +0000 UTC
2019/12/10 18:47:57 Found item in 'atdatasets': Data/20190821 VyA LAGUNA/Ground/DSC01812.JPG: 2019-08-26 10:23:01 +0000 UTC
2019/12/10 18:47:57 Found item in 'atdatasets': Data/20190821 VyA LAGUNA/Ground/DSC01813.JPG: 2019-08-26 10:23:01 +0000 UTC
2019/12/10 18:47:58 Found item in 'atdatasets': Data/: 2019-08-22 13:36:36 +0000 UTC
2019/12/10 18:47:58 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/: 2019-08-22 13:40:50 +0000 UTC
2019/12/10 18:47:59 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/Ground.tar.gz: 2019-08-22 13:58:36 +0000 UTC
2019/12/10 18:47:59 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/LOGS.tar.gz: 2019-08-22 14:00:34 +0000 UTC
2019/12/10 18:48:00 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/Mutispectral.tar.gz: 2019-08-26 07:46:38 +0000 UTC
2019/12/10 18:48:00 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/Thermal.tar.gz: 2019-08-23 07:21:39 +0000 UTC
2019/12/10 18:48:00 Found item in 'atdatasets': Data/20190821 VyA LAGUNA/: 2019-08-23 10:06:10 +0000 UTC
2019/12/10 18:48:01 Found item in 'atdatasets': Data/20190821 VyA LAGUNA/Ground/: 2019-08-26 10:22:55 +0000 UTC
2019/12/10 18:48:01 Found item in 'atdatasets': Data/20190821 VyA LAGUNA/Ground/DSC01812.JPG: 2019-08-26 10:23:01 +0000 UTC
2019/12/10 18:48:02 Found item in 'atdatasets': Data/20190821 VyA LAGUNA/Ground/DSC01813.JPG: 2019-08-26 10:23:01 +0000 UTC
2019/12/10 18:48:03 Found item in 'atdatasets': Data/: 2019-08-22 13:36:36 +0000 UTC
2019/12/10 18:48:04 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/: 2019-08-22 13:40:50 +0000 UTC
2019/12/10 18:48:04 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/Ground.tar.gz: 2019-08-22 13:58:36 +0000 UTC
2019/12/10 18:48:05 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/LOGS.tar.gz: 2019-08-22 14:00:34 +0000 UTC
2019/12/10 18:48:05 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/Mutispectral.tar.gz: 2019-08-26 07:46:38 +0000 UTC
2019/12/10 18:48:05 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/Thermal.tar.gz: 2019-08-23 07:21:39 +0000 UTC
2019/12/10 18:48:06 Found item in 'atdatasets': Data/20190821 VyA LAGUNA/: 2019-08-23 10:06:10 +0000 UTC
2019/12/10 18:48:07 Found item in 'atdatasets': Data/20190821 VyA LAGUNA/Ground/: 2019-08-26 10:22:55 +0000 UTC
2019/12/10 18:48:07 Found item in 'atdatasets': Data/20190821 VyA LAGUNA/Ground/DSC01812.JPG: 2019-08-26 10:23:01 +0000 UTC
2019/12/10 18:48:08 Found item in 'atdatasets': Data/20190821 VyA LAGUNA/Ground/DSC01813.JPG: 2019-08-26 10:23:01 +0000 UTC
2019/12/10 18:48:09 Found item in 'atdatasets': Data/: 2019-08-22 13:36:36 +0000 UTC
2019/12/10 18:48:09 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/: 2019-08-22 13:40:50 +0000 UTC
2019/12/10 18:48:10 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/Ground.tar.gz: 2019-08-22 13:58:36 +0000 UTC
2019/12/10 18:48:10 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/LOGS.tar.gz: 2019-08-22 14:00:34 +0000 UTC
2019/12/10 18:48:11 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/Mutispectral.tar.gz: 2019-08-26 07:46:38 +0000 UTC
2019/12/10 18:48:11 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/Thermal.tar.gz: 2019-08-23 07:21:39 +0000 UTC
2019/12/10 18:48:12 Found item in 'atdatasets': Data/20190821 VyA LAGUNA/: 2019-08-23 10:06:10 +0000 UTC
2019/12/10 18:48:12 Found item in 'atdatasets': Data/20190821 VyA LAGUNA/Ground/: 2019-08-26 10:22:55 +0000 UTC
2019/12/10 18:48:12 Found item in 'atdatasets': Data/20190821 VyA LAGUNA/Ground/DSC01812.JPG: 2019-08-26 10:23:01 +0000 UTC
2019/12/10 18:48:13 Found item in 'atdatasets': Data/20190821 VyA LAGUNA/Ground/DSC01813.JPG: 2019-08-26 10:23:01 +0000 UTC
2019/12/10 18:48:14 Found item in 'atdatasets': Data/: 2019-08-22 13:36:36 +0000 UTC
2019/12/10 18:48:14 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/: 2019-08-22 13:40:50 +0000 UTC
2019/12/10 18:48:14 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/Ground.tar.gz: 2019-08-22 13:58:36 +0000 UTC
2019/12/10 18:48:15 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/LOGS.tar.gz: 2019-08-22 14:00:34 +0000 UTC
2019/12/10 18:48:15 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/Mutispectral.tar.gz: 2019-08-26 07:46:38 +0000 UTC
2019/12/10 18:48:16 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/Thermal.tar.gz: 2019-08-23 07:21:39 +0000 UTC
2019/12/10 18:48:16 Found item in 'atdatasets': Data/20190821 VyA LAGUNA/: 2019-08-23 10:06:10 +0000 UTC
2019/12/10 18:48:17 Found item in 'atdatasets': Data/20190821 VyA LAGUNA/Ground/: 2019-08-26 10:22:55 +0000 UTC
2019/12/10 18:48:18 Found item in 'atdatasets': Data/20190821 VyA LAGUNA/Ground/DSC01812.JPG: 2019-08-26 10:23:01 +0000 UTC
2019/12/10 18:48:18 Found item in 'atdatasets': Data/20190821 VyA LAGUNA/Ground/DSC01813.JPG: 2019-08-26 10:23:01 +0000 UTC
2019/12/10 18:48:20 Found item in 'atdatasets': Data/: 2019-08-22 13:36:36 +0000 UTC
2019/12/10 18:48:20 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/: 2019-08-22 13:40:50 +0000 UTC
2019/12/10 18:48:20 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/Ground.tar.gz: 2019-08-22 13:58:36 +0000 UTC
2019/12/10 18:48:21 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/LOGS.tar.gz: 2019-08-22 14:00:34 +0000 UTC
2019/12/10 18:48:21 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/Mutispectral.tar.gz: 2019-08-26 07:46:38 +0000 UTC
2019/12/10 18:48:22 Found item in 'atdatasets': Data/20190813 VyA LAGUNA/Thermal.tar.gz: 2019-08-23 07:21:39 +0000 UTC
2019/12/10 18:48:22 Found item in 'atdatasets': Data/20190821 VyA LAGUNA/: 2019-08-23 10:06:10 +0000 UTC
2019/12/10 18:48:23 Found item in 'atdatasets': Data/20190821 VyA LAGUNA/Ground/: 2019-08-26 10:22:55 +0000 UTC
2019/12/10 18:48:23 Found item in 'atdatasets': Data/20190821 VyA LAGUNA/Ground/DSC01812.JPG: 2019-08-26 10:23:01 +0000 UTC
2019/12/10 18:48:24 Found item in 'atdatasets': Data/20190821 VyA LAGUNA/Ground/DSC01813.JPG: 2019-08-26 10:23:01 +0000 UTC
panic: too many items: 100

goroutine 1 [running]:
main.main()
        C:/Users/***/go/src/***/s3test/main.go:60 +0x493
exit status 2
@inliquid
Copy link
Author

One important note: if pageSize> than a total items in container, function works as expected. But if it's < than number of items, it results in infinite loop.

@jasonsattler
Copy link
Contributor

Based on your output looks like the start cursor is not working. Which we depend on that to get the next set of results. Can you check to see if that api fully support S3.ListObjectsV2
https://github.com/graymeta/stow/blob/master/s3/container.go#L48-L55

@inliquid
Copy link
Author

Hi, yes, I came up to actually use AWS SDK and call ListObjectsV2 directly which seems to work perfectly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@inliquid @jasonsattler