Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(evm): collaborative indexing through txid sharing #3833

Merged
merged 20 commits into from
Jun 7, 2024

Conversation

daniel-savu
Copy link
Contributor

@daniel-savu daniel-savu commented May 21, 2024

Description

  • improved indexing reliability by sharing channels between indexing tasks, such that they inform one another when they come across a hyperlane-related transaction
  • the receiving end will then get the receipt of that tx and filter it for events relevant to it
  • this adds some flexibility to indexing since we can stop relying on watermark cursor entirely: as long as we have one task using sequence indexing, all others can rely on being informed by their channels about what txs to index
  • this PR is only for the EVM chains, but can be easily extended

Drive-by changes

  • Enforces gas payments for all chains in e2e (previously this only used to be the case for sealevel)
  • The channel size is configured per type, similarly to the indexing cursor, in contract_sync/cursors/mod.rs
  • Reduces log verbosity is some places, e.g. by manually implementing Debug to exclude fields that aren't relevant to debugging (maybe there's even a macro annotation to exclude, but I haven't checked)
  • Renames MpmcReceiver -> BroadcastReceiver

Related issues

Backward compatibility

Testing

e2e tested with the watermark cursor manually disabled

Copy link

changeset-bot bot commented May 21, 2024

⚠️ No Changeset found

Latest commit: 69dd7e1

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

Copy link

codecov bot commented May 21, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 62.76%. Comparing base (d10b035) to head (69dd7e1).
Report is 7 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #3833   +/-   ##
=======================================
  Coverage   62.76%   62.76%           
=======================================
  Files         125      125           
  Lines        1579     1579           
  Branches      175      175           
=======================================
  Hits          991      991           
  Misses        561      561           
  Partials       27       27           
Components Coverage Δ
core 84.61% <ø> (ø)
hooks 67.50% <ø> (ø)
isms 39.52% <ø> (ø)
token 60.83% <ø> (ø)
middlewares 77.05% <ø> (ø)

@daniel-savu daniel-savu marked this pull request as ready for review May 24, 2024 13:12
@daniel-savu daniel-savu changed the title feat: collaborative indexing through txid sharing feat(evm): collaborative indexing through txid sharing May 24, 2024
Copy link
Collaborator

@tkporter tkporter left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice

rust/agents/relayer/src/relayer.rs Outdated Show resolved Hide resolved
rust/agents/relayer/src/relayer.rs Outdated Show resolved Hide resolved
rust/hyperlane-base/src/contract_sync/mod.rs Outdated Show resolved Hide resolved
rust/agents/relayer/src/relayer.rs Outdated Show resolved Hide resolved
rust/agents/relayer/src/relayer.rs Outdated Show resolved Hide resolved
rust/agents/scraper/src/agent.rs Outdated Show resolved Hide resolved
rust/hyperlane-base/src/contract_sync/mod.rs Outdated Show resolved Hide resolved
rust/agents/relayer/src/relayer.rs Outdated Show resolved Hide resolved
rust/agents/relayer/src/relayer.rs Outdated Show resolved Hide resolved
rust/hyperlane-base/src/contract_sync/mod.rs Outdated Show resolved Hide resolved
Copy link
Collaborator

@tkporter tkporter left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving but I believe #3833 (comment) may need to be changed before merging

rust/utils/run-locally/src/main.rs Show resolved Hide resolved
@daniel-savu daniel-savu added this pull request to the merge queue Jun 7, 2024
Merged via the queue into main with commit 942aba8 Jun 7, 2024
38 checks passed
@daniel-savu daniel-savu deleted the dan/hook-indexing branch June 7, 2024 12:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Hook indexing learns about new messages as they are indexed
2 participants