Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Race top n nodes when making requests #25

Merged
merged 40 commits into from
Oct 26, 2023
Merged

Conversation

AmeanAsad
Copy link
Contributor

@AmeanAsad AmeanAsad commented Oct 24, 2023

Description

  • Add a race between nodes when fetching a cid. There is a limit set for how many nodes to race at once. Currently it is at 3

AmeanAsad and others added 30 commits October 11, 2023 00:17
Co-authored-by: Diego Rodríguez Baquero <diego@protocol.ai>
* Abort on error (#19)

* feat: use controller from options if exists. abort fetch if error occurs.

* test: check if external abort controller is used

* build: move build output to dist/ folder

* fix: newline

* 0.1.1

* Build exports (#20)

* chore: rename file

* feat: add new entrypoint with exports.

Switch Saturn to named export

* build: expose entire module instead of just the default export

* docs: update README

* 0.2.0

* feat: include worker scopes when checking for browser runtime (#21)

* 0.3.0

---------

Co-authored-by: Eric Guan <guanzo91@gmail.com>
@AmeanAsad AmeanAsad marked this pull request as ready for review October 24, 2023 18:11
src/client.js Outdated
const controllers = []

const createFetchPromise = async (origin) => {
options.url = origin
Copy link
Collaborator

@guanzo guanzo Oct 24, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better for each function call to make a copy of the options object instead of sharing it and mutating fields.

Suggested change
options.url = origin
const fetchOptions = { ...options, url: origin } // need to rename the rest of the references

This is only a shallow copy but should suffice.

src/client.js Outdated Show resolved Hide resolved
src/client.js Outdated Show resolved Hide resolved
src/client.js Outdated Show resolved Hide resolved
@AmeanAsad AmeanAsad changed the title Feat/race nodes feat: Race top n nodes when making requests Oct 24, 2023
if (fallbackCount > this.opts.fallbackLimit) {
return
}
opts.url = origin.url
if (opts.raceNodes) {
const origins = nodes.slice(i, i + Saturn.defaultRaceCount).map((node) => node.url)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So if we have nodes a,b,c,d,e, then the race groups are:

  • a,b,c
  • 1st fallback: b,c,d
  • 2nd fallback: c,d,e

Is that right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is correct. Will also note that this is not necessarily the best way to do it and has some obvious room for improvement. For example:

  • If node c fails, it we might hit it twice again before another good fallback is contacted.

I think as a next step, the best way to do this is to remove nodes that fail and such that:

  • initial group is a, b, c. Assume node c fails
  • 1st fallback is a, b, d and so on

I think this can be better achieved with a hashring implementation but we can re-evaluate.

@AmeanAsad AmeanAsad merged commit 3b4e8db into main Oct 26, 2023
1 check passed
@AmeanAsad AmeanAsad deleted the feat/race-nodes branch October 26, 2023 19:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants