deploy: 81f632d

previews/44/.buildinfo

@@ -0,0 +1,4 @@
+# Sphinx build info version 1
+# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
+config: 5b25d5cf7d93ce683eca3959b9565354
+tags: 645f666f9bcd5a90fca523b33c5a78b7

previews/44/_sources/api_doc/api.rst

@@ -0,0 +1,7 @@
+Internal Django
+===============
+
+.. The following text is only a placeholder in case the api can not be
+   generated (e.g. for the pdf version)
+
+The API Documentation is available at https://jonas-rem.github.io/lwm2m_server.

previews/44/_sources/architecture/communication_interfaces.rst

@@ -0,0 +1,164 @@
+Communication, Interfaces
+==========================
+
+The communication between IoT devices and Leshan is specified by the OMA LwM2M
+standard:
+
+- `LwM2M core specification v1.1.1`_
+- `LwM2M transport binding v1.1.1`_
+
+.. _LwM2M core specification v1.1.1: https://www.openmobilealliance.org/release/LightweightM2M/V1_1_1-20190617-A/OMA-TS-LightweightM2M_Core-V1_1_1-20190617-A.pdf
+.. _LwM2M transport binding v1.1.1: https://www.openmobilealliance.org/release/LightweightM2M/V1_1_1-20190617-A/OMA-TS-LightweightM2M_Transport-V1_1_1-20190617-A.pdf
+
+The standard describes how the LwM2M server (Leshan) works, however, it does
+not describe how to connect a backend server to Leshan. The backend is
+responsible for storing the data in a database and implementing application
+logic. A frontend can access the data in the database and visualize outward
+facing user interfaces. Leshan acts as a gateway between Endpoints and the
+backend. There should be no application specific logic implemented in Leshan.
+
+In order to communicate and exchange data, both components (Leshan LwM2M Server
+and Django) post data to each other's ReST APIs. Communication is typically
+triggered by IoT devices sending data or the user/application requesting data
+from devices.
+
+Data Flow: Endpoint -> Backend
+------------------------------
+
+All communication from Endpoints to the server flows through Leshan. Leshan
+interprets this data according to the LwM2M protocol and generates a ReST API
+call to the backend server. The backend server then stores the data in the
+database.
+
+Django hosts the ReST API that Leshan posts to. Serializers, a part of Django,
+deserialize the incoming data and store it in the database according to the
+database model.
+
+Leshan Data Format
+..................
+
+There are two types of data that Leshan sends to the backend, single resource
+and composite resource format. The two ReST API endpoints that Leshan posts to
+are available under the following URLs:
+
+- ``/leshan_api/resource/single``: single resource format.
+- ``/leshan_api/resource/composite``: composite resource format.
+
+For more details, please theck the ``Internal Django`` API documentation.
+
+.. code-block:: json
+  :caption: Single Resource Format (3303/0/5700)
+
+  {
+     "ep": "qemu_x86",
+     "obj_id": 3303,
+     "val": {
+         "kind": "singleResource",
+         "id": 5700,
+         "type": "FLOAT",
+         "value": "24.899181214836236"
+     }
+  }
+
+.. code-block:: json
+  :caption: Composite Resource Format (3/0/0..17)
+  :emphasize-lines: 8,26,23
+
+  {
+      "ep" : "qemu_x86",
+      "val" : {
+          "instances" : [ {
+              "kind" : "instance",
+              "resources" : [ {
+                  "kind" : "singleResource",
+                  "id" : 0,
+                  "type" : "STRING",
+                  "value" : "Zephyr"
+              }, {
+                  "kind" : "multiResource",
+                  "values" : {
+                      "0" : "1",
+                      "1" : "5"
+                  },
+              },
+                  "kind" : "singleResource",
+                  "id" : 17,
+                  "type" : "STRING",
+                  "value" : "qemu_x86"
+              } ],
+              "id" : 0
+          } ],
+          "kind" : "obj",
+          "id" : 3
+      }
+  }
+
+The marked lines in the composite resource format show where Object ID,
+Instance ID and Resource ID are located. A composite resource format can
+consist of multiple Object IDs.
+
+.. warning::
+
+  Currently multiResources are not supported and will be ignored. MultiResource
+  dataypes  are e.g. used for Voltage range (min, max).
+
+Registration Updates
+....................
+
+Data Flow: Backend -> Endpoint
+------------------------------
+
+Endpoints often operate in queue mode, meaning they are not always online.
+The LwM2M Server is aware of the current status of a device (Online/Offline)
+and communicates this status to the backend server. Leshan does not queue
+pending data that should be sent to the device when it comes online. The
+backend server must handle this by itself so it has to have a representation of
+the current status of each device as well as the data to be send. The resource
+table ``DeviceOperation`` is used to store pending operations that should be
+sent to the endpoint while it is online.
+
+Once an endpoint updates it's registration (LwM2M Update Operation) Leshan
+notifies the backend. The backend checks the ``DeviceOperation`` table for
+pending operations and sends them to the device by posting to the Leshan hosted
+ReST API. Leshan keeps the post call open until the device acknowledges the
+operation or a timeout is generated. Endpoints can be slow to respond (several
+Seconds), so the backend has to handle the ReST API call in an asynchronous
+manner. By only sending data to endpoints while they are online, the backend
+can be sure that the ReST API calls are not open for a long time.
+
+Asynchronous Communication
+--------------------------
+
+Given that endpoints are comparably slow to respond, handling communication
+asynchronously is essential for efficient operation. This can be effectively
+managed using Celery, a distributed task queue. When Leshan notifies the
+backend of an endpoint status update, Celery can be used to handle the
+long-running API calls, ensuring that the backend remains responsive and
+scalable. As the backend communicates with many endpoints simultaneously, an
+efficient queing mechanism is essential to ensure that the system remains
+responsive and scalable.
+
+Before the backend executes the API call, it updates the endpointOperation
+status to ``SENDING``, indicating an ongoing operation.
+
+Once the API call is complete the database will be updated with the result
+(e.g. ``CONFIRMED``, ``FAILED``, ``QUEUED``) depending on the result of the
+request. The ``FAILED`` status is assigned after 3 attempts. Retransmissions are
+triggered when the endpoint updates it's registration the next time.
+
+Example Communication
+---------------------
+
+The following example shows how the backend server can send a firmware download
+link resource ``Package URI 5/0/1`` to an endpoint:
+
+#. User creates new ``DeviceOperation``: resource path ``5/0/1``, value
+   ``https://url.com/fw.bin``.
+#. Backend checks endpoint online status.
+#. If endpoint is offline, no further action is taken right away.
+#. Endoint comes online, Leshan sends update to the backend.
+#. Backend checks ``DeviceOperation`` table for pending operations for the
+   endpoint.
+#. Finds pending operation, send resource to endpoint via the Leshan ReST API.
+#. Pending operation is marked ``completed`` if the endpoint acknowledges the
+   operation.

previews/44/_sources/architecture/django.rst

@@ -0,0 +1,118 @@
+Django
+======
+
+The Django server hosts the REST API, manages the database, and provides a web interface
+for data visualization and user management. This server acts as the backend
+infrastructure that supports the entire IoT ecosystem by:
+
+* **REST API**: Facilitates communication between the IoT devices and the server.
+  The API endpoints handle requests for data uploads from devices, send control commands,
+  and provide access to stored telemetry data.
+
+* **Database Management**: Stores all the telemetry data, device information, user
+  accounts, and configuration settings. The database ensures that data is organized,
+  searchable, and retrievable in an efficient manner.
+
+* **Web Interface**: Offers a user-friendly interface for data visualization and
+  management. Through this interface, users can monitor device status, visualize
+  telemetry data in real-time, manage user accounts, and configure device settings.
+
+
+Build and Run
+-------------
+
+The Django server can also run locally, without the need of a docker container.
+Make sure to create a virtual environment and install the requirements:
+
+.. code-block:: console
+
+  host:~$ source venv/bin/activate
+  host:~$ cd flownexus_workspace/lwm2m_server/server/django
+  host:lwm2m_server/server/django$ pip install -r requirements.txt
+  host:lwm2m_server/server/django$ ./django_start.sh
+
+The Django server should now be up and running under the following URL:
+``http://localhost:8000/admin``. The admin login is ``admin`` and the password
+
+
+Run Unit Tests
+..............
+
+There are unit tests, that test the deserializer, which parses the Json payload
+from the ReST API. You can run the unit tests with the following command:
+
+.. code-block:: console
+
+  host:~/flownexus_workspace/lwm2m_server/server/django$ python manage.py test sensordata
+  Found 2 test(s).
+  Creating test database for alias 'default'...
+  ----------------------------------------------------------------------
+  Ran 2 tests in 0.008s
+
+  OK
+  Destroying test database for alias 'default'...
+
+
+Database Model
+--------------
+
+The database model is the core of the Django server. It aims to store
+information according to the LwM2M resource model. The advantage is that data
+can be stored in a generic way and the server can be extended with new
+resources without changing the database schema.
+
+The server application logic only has to handle higher level events. Those
+events are situations where multiple resources are associated (e.g.
+Temperature, Pressure, Humidity, Acceleration). Those multiple resources are
+linked in the event, together with a timestamp. The event itself is represented
+by the database model in a generic way, several custom event types can be
+created by the application logic.
+
+An Entity Relationship Diagram (ERD) is a visual representation of the database
+schema. It is automatically generated from the Django models. ``sensordata`` is
+the Django app that contains the application logic.
+
+.. figure:: ../images/erd.svg
+
+  Entity Relationship Diagram generated from Django models
+
+.. glossary::
+
+  Endpoint
+    IoT devices using the LwM2M protocol in the network, identifiable by a
+    unique name, e.g. ``urn:imei:123456789012345``.
+
+  ResourceType
+    Defines resource data points comprehensively, annotating each with a unique
+    object-resource ID combination, a descriptive name, and specifying the
+    expected data type.
+
+  Resource
+    Captures individual data values from IoT devices, annotated with
+    timestamps, applicable data types, and linked to both the device and
+    resource type for which the data pertains.
+
+  Event
+    Serves as a collection point for significant occurrences reported by
+    devices, including composite events defined by enclosing object IDs. The
+    server application logic has to generate events based on matching
+    timestamps or received composite notifications from devices. Although
+    individual resources within an event may have different timestamps, the
+    event itself encapsulates a single timestamp.
+
+  EventResource
+    Acts as a junction table forming a many-to-many relationship between events
+    and their constituent resources, enabling flexible association without
+    direct modification to the core events or resources tables.
+
+  EndpointOperation
+    Represents actionable commands or processes targeted at endpoints, tracking
+    the operation type, status, and scheduling through timestamps, also
+    detailing the transmission attempts and last action.
+
+  Firmware
+    Stores metadata about firmware binaries that are available for devices to
+    download and install. Each record includes a version identifier, the name
+    of the file, a URL from where the device can retrieve the firmware, and
+    timestamps for tracking when each firmware record was created and last
+    updated.

previews/44/_sources/architecture/index.rst

@@ -0,0 +1,11 @@
+Architecture
+============
+
+This section describes the architecture of individual components and how they
+interact with each other.
+
+.. toctree::
+
+   django
+   leshan
+   communication_interfaces

previews/44/_sources/architecture/leshan.rst

@@ -0,0 +1,18 @@
+Leshan
+======
+
+Build and Run
+-------------
+
+The Leshan server can also run locally, without the need of a docker container.
+
+.. code-block:: console
+
+  host:~$ sudo apt update
+  host:~$ sudo apt install openjdk-17-jdk maven
+  host:~$ source venv/bin/activate
+  host:~$ cd flownexus_workspace/lwm2m_server/server/leshan
+  host:lwm2m_server/server/leshan$ ./leshan_build_run.sh
+
+The Leshan server should now be up and running under the following URL:
+``http://localhost:8080``.