Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support multiple UDP source ports (multiport) #768

Open
wants to merge 16 commits into
base: master
Choose a base branch
from
Open

Conversation

wadey
Copy link
Member

@wadey wadey commented Oct 17, 2022

The goal of this work is to send packets between two hosts using more than one 5-tuple. When running on networks like AWS where the underlying network driver and overlay fabric makes routing, load balancing, and failover decisions based on the flow hash, this enables more than one flow between pairs of hosts.

Multiport spreads outgoing UDP packets across multiple UDP send ports, which allows nebula to work around any issues on the underlay network. Some example issues this could work around:

  • UDP rate limits on a per flow basis.
  • Partial underlay network failure in which some flows work and some don't

Agreement is done during the handshake to decide if multiport mode will be used for a given tunnel (one side must have tx_enabled set, the other side must have rx_enabled set)

NOTE: you cannot use multiport on a host if you are relying on UDP hole punching to get through a NAT or firewall.

NOTE: Linux only (uses raw sockets to send). Also currently only works with IPv4 underlay network remotes.

This is implemented by opening a raw socket and sending packets with a source port that is based on a hash of the overlay source/destiation port. For ICMP and Nebula metadata packets, we use a random source port.

Example configuration:

multiport:
  # This host support sending via multiple UDP ports.
  tx_enabled: false

  # This host supports receiving packets sent from multiple UDP ports.
  rx_enabled: false

  # How many UDP ports to use when sending. The lowest source port will be
  # listen.port and go up to (but not including) listen.port + tx_ports.
  tx_ports: 100

  # NOTE: All of your hosts must be running a version of Nebula that supports
  # multiport if you want to enable this feature. Older versions of Nebula
  # will be confused by these multiport handshakes.
  #
  # If handshakes are not getting a response, attempt to transmit handshakes
  # using random UDP source ports (to get around partial underlay network
  # failures).
  tx_handshake: false

  # How many unresponded handshakes we should send before we attempt to
  # send multiport handshakes.
  tx_handshake_delay: 2

The goal of this work is to send packets between two hosts using more than one
5-tuple. When running on networks like AWS where the underlying network driver
and overlay fabric makes routing, load balancing, and failover decisions based
on the flow hash, this enables more than one flow between pairs of hosts.

Multiport spreads outgoing UDP packets across multiple UDP send ports,
which allows nebula to work around any issues on the underlay network.
Some example issues this could work around:

- UDP rate limits on a per flow basis.
- Partial underlay network failure in which some flows work and some don't

Agreement is done during the handshake to decide if multiport mode will
be used for a given tunnel (one side must have tx_enabled set, the other
side must have rx_enabled set)

NOTE: you cannot use multiport on a host if you are relying on UDP hole
punching to get through a NAT or firewall.

NOTE: Linux only (uses raw sockets to send). Also currently only works
with IPv4 underlay network remotes.

This is implemented by opening a raw socket and sending packets with
a source port that is based on a hash of the overlay source/destiation
port. For ICMP and Nebula metadata packets, we use a random source port.

Example configuration:

    multiport:
      # This host support sending via multiple UDP ports.
      tx_enabled: false

      # This host supports receiving packets sent from multiple UDP ports.
      rx_enabled: false

      # How many UDP ports to use when sending. The lowest source port will be
      # listen.port and go up to (but not including) listen.port + tx_ports.
      tx_ports: 100

      # NOTE: All of your hosts must be running a version of Nebula that supports
      # multiport if you want to enable this feature. Older versions of Nebula
      # will be confused by these multiport handshakes.
      #
      # If handshakes are not getting a response, attempt to transmit handshakes
      # using random UDP source ports (to get around partial underlay network
      # failures).
      tx_handshake: false

      # How many unresponded handshakes we should send before we attempt to
      # send multiport handshakes.
      tx_handshake_delay: 2
@wadey wadey added this to the v1.7.0 milestone Oct 17, 2022
@wadey wadey mentioned this pull request Oct 17, 2022
@brad-defined
Copy link
Collaborator

Branch now has conflicts, needs updating

@wadey wadey added needs-defined-net-review Review needed from a Defined Networking team member needs-slack-review Needs review from a Slack team member labels Mar 13, 2023
@nbrownus nbrownus modified the milestones: v1.7.0, v1.8.0 Apr 3, 2023
@nbrownus nbrownus modified the milestones: v1.8.0, v1.9.0 Oct 30, 2023
@nbrownus nbrownus modified the milestones: v1.9.0, v1.10.0 Apr 22, 2024
wadey added 4 commits May 8, 2024 11:22
We need to compare just the IPs here and not the IP+Port

This is a regression with the merge of v1.9.4 and the change to
netip.AddrPort

- dabce8a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs-defined-net-review Review needed from a Defined Networking team member needs-slack-review Needs review from a Slack team member slack
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants