Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MQTT subscription #361

Open
wants to merge 11 commits into
base: master
Choose a base branch
from
Open

MQTT subscription #361

wants to merge 11 commits into from

Conversation

timcowlishaw
Copy link
Contributor

@timcowlishaw timcowlishaw commented Sep 25, 2024

Addresses #355, allowing us to scale the mqtt task through parallel processing. We have two named "main" subscribers which ensure that messages are not dropped in the event of downtime (they have consistent client ids and persistent sessions), and a secondary subscriber role (with transient ids and clean sessions) which can be scaled with load. Needs thorough testing on staging before deployment!

⚠️ on deployment, we need to: ⚠️

  • bring down the app entirely, including the old mqtt-task
  • delete the old "mqtt subscriber" client in emqx, so that no messages are retained for it, as the client id changes in this release
  • run migrations BEFORE bringing up the mqtt-task again, as otherwise duplicate components will be created
  • bring up the entire stack again

@timcowlishaw timcowlishaw linked an issue Sep 25, 2024 that may be closed by this pull request
6 tasks
@timcowlishaw
Copy link
Contributor Author

(deployed to staging on branch staging-2024-01-01, although it was actually deployed on the 25th of sept, sorry for the confusion)

@timcowlishaw
Copy link
Contributor Author

#TODO: ensure that mqtt-task gets an exclusive lock on the components table before creating a component - it only happens in "unrealistic" testing conditions currently, but it is possible for two identical components to be created at once

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Log a sentry error when the MQTT internal queue reaches saturation point Shared subscription in MQTT
2 participants