Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

List all topics using pagination #39

Open
Arkenan opened this issue Mar 15, 2022 · 1 comment
Open

List all topics using pagination #39

Arkenan opened this issue Mar 15, 2022 · 1 comment

Comments

@Arkenan
Copy link

Arkenan commented Mar 15, 2022

Currently there's a limit of about 100 topics when calling Kane.Topic.all function, as the pubsub API paginates the results. There are several ways of removing this 100 topic limit, but the easiest one is, by default, traversing the next pages.

When listing topics, we get a map of the following kind:

%{
  "topics" => list(%Kane.Topic{}),
  "nextPageToken" => str
}

We can GET projects/#{project}/topics?pageToken=#{page_token} to get the next page. If we use this recursively we can get all of the tokens until the map no longer has the nextPageToken present and we conclude that's the last page.

@Arkenan
Copy link
Author

Arkenan commented Mar 15, 2022

Here's an example PR of how we can do this: #40

We can proceed on that idea in that same PR or we can move to a different one, I'm open to discuss ideas! That PR was tested with a pubsub with more than 100 topics and it worked without any issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant