db: Update generic database model #8
Conversation
jonas-rem
force-pushed
the
WIP/jonas-rem/update_db_model
branch
2 times, most recently
from
a7f9f2a
to
9c7bb44
Compare
jonas-rem
changed the title
jonas-rem
changed the title
|
Those are example screenshots from a local django instance with added resources from simulated Zephyr devices.
|
jonas-rem
changed the title
|
@Kappuccino111 I added a new instance of the database where this model has been applied. In this database there are the above shown values as a demonstration. The access is admin/password that can be used for development and testing. |
jonas-rem
force-pushed
the
WIP/jonas-rem/update_db_model
branch
2 times, most recently
from
5e3e335
to
6199930
Compare
There was a problem hiding this comment.
@jonas-rem I seem to have got the basics of the above database clear 😁🙌🏻. The approach for handling data with Resource, ResourceType, and Event tables definitely allows for flexibility and extensibility, since now we can add new types of IoT data without modifying the database schema and allows for an event driven architecture.
But here are some bottlenecks I think we could face :
Complex querying : What if we need to generate a report that includes data from temperature sensors, humidity sensors, and pressure sensors across multiple devices? This query could involve multiple self-joins on the resources table to combine different data types, which can become highly complex and inefficient as the number of resources and devices increases.
Performance Issue : Since we are also targeting a realtime graph dashboard we would need quick realtime updates. As data grows the response time slows down
Data Integrity : The current db does rely heavily on core application logic. If the application fails to correctly manage the mapping of these new data points to the existing ResourceType entities due to some bug, it could lead to incorrect data being stored and represented. Also debugging can become a bit difficult.
Let me know WDYT??
My primary reasoning was to have a normalized generic database. Adding data should be easy, as this is happening more often than searching for data. Think about 10 k IoT devices which send data to the server regularly, compared to a few dashboards occasionally running. Further I assumed that events (aggregation of several resources) may be more important to visualize in a dashboard than comparing resources like temperature values. But we can discuss these assumptions and optimize the model. Do you see a way to improve performance without loosing the current advantages (normalized, generic, no conversions)?
The basic assignments are specified by the LwM2M Object and Resource IDs. If custom resources are created, those must not change, once defined. Also the application does not implement a mapping or something like that, the resource and object IDs are directly coming from firmware. |
jonas-rem
force-pushed
the
WIP/jonas-rem/update_db_model
branch
from
6199930
to
2484086
Compare
The generic model uses one table for all possible lwm2m resources. The resources can be accessed through their resource_id and object_id according to the L2M2M Specification. This removes the need for a manual mapping of resource/object_id to table names like temperature. Signed-off-by: Jonas Remmert <jremmert@gmx.net>
- Rewrite the serializer for single and multi resource - Adapt admin dashboard - Remove listView for now, admin dashboard is sufficient for testing currently Signed-off-by: Jonas Remmert <jremmert@gmx.net>
- delete actual database from repository and add initial json data for registering a admin user. Database will be automatically created on first start. - Admin login is admin/password for easy testing. - There is one set of boot message already stored in the database as a showcase. Signed-off-by: Jonas Remmert <jremmert@gmx.net>
Signed-off-by: Jonas Remmert <jremmert@gmx.net>
Load initial data when creating a new database. Do not update static files at each start in the django startup script. Signed-off-by: Jonas Remmert <jremmert@gmx.net>
Signed-off-by: Jonas Remmert <jremmert@gmx.net>
jonas-rem
force-pushed
the
WIP/jonas-rem/update_db_model
branch
from
2484086
to
709df45
Compare
|
@jonas-rem I think the base implementation looks good to go for now. We can make further improvements as needed later and proceed to merge this PR. 👍😁 |
|
@jonas-rem rebased and merged the PR 👍 |
WIP: 9192426 is the relevant commit that drafts a database model. a7f9f2a provides an explanation.
Based on #6 (Entity Relationship Diagram Generation)
The heart of the application is the database model. This PR proposes a database model that is generic and can be used for all LwM2M resources. This simplifies the database and allows to serialize incoming messages automatically without any LwM2M Object_ID/Resource_ID mapping.
https://techlibrary.hpe.com/docs/otlink-wo/OMA-LWM2M-Object-Resource-and-Value-Details.html gives an overview how resources are specified in LwM2M.