Asynchronous communication with Endoints via celery tasks #37
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jonas-rem
changed the title
jonas-rem
changed the title
jonas-rem
force-pushed
the
WIP/jonas-rem/async_comm
branch
from
ba214fc
to
be65a11
Compare
This was referenced Jun 14, 2024
Kappuccino111
force-pushed
the
WIP/jonas-rem/leshan_registration_handler
branch
from
d1ff295
to
eddf65f
Compare
Base automatically changed from
WIP/jonas-rem/leshan_registration_handler
to
main
July 4, 2024 20:45
Integrate Celery task queue. Redis serves as the message broker for the Celery task queue. Redis can run in a seperate container with of-the-shelf available images. Gevent is used an efficient alternative compared to multiple python processes, where each process has to load it's own runtime resulting in high memory requirement. Celery is used for functions that wait for ReST API calls to Leshan, an IO-bound task. Currently, Celery starts 1 worker (1 CPU), however, this worker is sufficient to handle more than 100 simultaneous ReST API calls efficiently. Signed-off-by: Jonas Remmert <jremmert@gmx.net>
Signed-off-by: Jonas Remmert <jremmert@gmx.net>
Signed-off-by: Jonas Remmert <jremmert@gmx.net>
The asynchronous function process_pending_operationss checks the database for queued endpointOperations and sends them to the related endpoint. The function is called e.g. during data insertion via the Admin dashboard or during an registration update of an endpoing. Signed-off-by: Jonas Remmert <jremmert@gmx.net>
Signed-off-by: Jonas Remmert <jremmert@gmx.net>
jonas-rem
force-pushed
the
WIP/jonas-rem/async_comm
branch
from
be65a11
to
abb3f7f
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Based on #34
Communicating with Endpoints through a ReST API that is provided by leshan is slow. There can be several scenarios where an endpoints takes time to acknowledge (slow network, offline Endpoint, Endpoint waits for next eDRX Interval). The ReST call stays open for this time, until a timeout (up to several min) occurs.
It is essential to handle those ReST API calls in an asynchronous way so that the Django app can operate efficiently. API calls that are executed asynchronously via Celery still have access to Djangos database model and can update the result in the database after completion.