Changing root topics names for clarity

README.md

@@ -27,16 +27,22 @@ The core functionality is supported by:
 
 ## Specifics
 
+### Data flow
+
+![](./data-flow.drawio.svg)
+
 ### Data format
 The hub have some (not a lot) of expectations on the data format that flows in the system:
 * The database is configured to use a narrow table setup according to
   ```
   time (TIMESTAMPZ)   |   vessel_id (TEXT)   |   parameter_id (TEXT)   |   value (TEXT)
   ```
-* The data ingestor is very flexible in its configuration about how to map data from the MQTT world to the database layout, see https://github.com/MO-RISE/pontos-data-ingestor#specifics. In essence, the only hard requirement is that the payloads are expected to be vaild `JSON`. 
+* The data ingestor is very flexible in its configuration about how to map data from the MQTT world to the database layout, see https://github.com/MO-RISE/pontos-data-ingestor#specifics. In essence, the only hard requirement is that the payloads are expected to be vaild `JSON`.
 
 #### Pontos project specific data format
-Within the Pontos project, a more specific data format has been agreed upon, the Pontos Data Format. This data format is described more in detail [here](https://github.com/MO-RISE/pontos-data-format). **NOTE:** The default configuration of the data ingestor setup for pontos hub is according to the Pontos Data Format.
+Within the Pontos project, a more specific data format has been agreed upon, the Pontos Data Format. This data format is described more in detail [here](https://github.com/MO-RISE/pontos-data-format).
+
+**NOTE:** The default configuration of pontos-hub is according to the Pontos Data Format.
 
 ### Authn / Authz
 The datahub is developed with the primary aim of being an open datahub where anyone can publicly access data. This, however, does not entirely remove the need for a software solution dealing with Authentication / Authorization, for the following anticipated reasons:
@@ -64,7 +70,7 @@ The MQTT API for the datahub is configured such that:
 * Authorization is enforced through access control lists (acl) that may be provided in two ways:
     * As a separate file on disk (see https://www.emqx.io/docs/en/v5.1/access-control/authz/file.html)
     * As part of the JWT `acl` claim (see https://www.emqx.io/docs/en/v5.1/access-control/authn/jwt.html#authorization-list-optional)
-* A default acl is bundled with the datahub, see [`acl.conf`](./broker/acl.conf) which gives read access to `PONTOS/#` for everyone.
+* A default acl is bundled with the datahub, see [`acl.conf`](./broker/acl.conf) which gives read access to `PONTOS_EGRESS/#` for everyone.
 
 
 #### Token generation for read access to the datahub (both via MQTT and REST APIs)
@@ -113,9 +119,9 @@ To generate a valid token for write access to the MQTT API there are two options
 * Use of in-JWT acl rules (i.e. define custom acl rules within the JWT itself)
   * Encode a new JWT using the following example as a guideline:
     ```
-    jwt encode --iss=pontos-hub --secret='<your-pontos-hub-JWT-secret>' --sub='<your-preferred-username>' '{"acl":{"pub":["PONTOS/<vessel_id>/#"]}}'
+    jwt encode --iss=pontos-hub --secret='<your-pontos-hub-JWT-secret>' --sub='<your-preferred-username>' '{"acl":{"pub":["PONTOS_INGRESS/<vessel_id>/#"]}}'
     ```
-    Which will allow `<your-preferred-username>` to publish data to topics matching `PONTOS/<vessel_id>/#`.
+    Which will allow `<your-preferred-username>` to publish data to topics matching `PONTOS_INGRESS/<vessel_id>/#`.
 
 
 ## Deploy

broker/acl.conf

@@ -1,10 +1,10 @@
-%% Allow access for internal users to PONTOS and PONTOS_HUB root topics
-{allow, {user, "__internal__"}, publish, ["PONTOS_HUB/#"]}.
-{allow, {user, "__internal__"}, subscribe, ["PONTOS/#"]}.
+%% Allow access for internal users to PONTOS_EGRESS and PONTOS_INGRESS root topics
+{allow, {user, "__internal__"}, publish, ["PONTOS_EGRESS/#"]}.
+{allow, {user, "__internal__"}, subscribe, ["PONTOS_INGRESS/#"]}.
 
-%% Allow subscriptions to the PONTOS root topic from anyone
+%% Allow subscriptions to the PONTOS_EGRESS root topic from anyone
 %% that is already authenticated
-{allow, all, subscribe, ["PONTOS_HUB/#"]}.
+{allow, all, subscribe, ["PONTOS_EGRESS/#"]}.
 
 %% Deny everything else
 {deny, all}.

docker-compose.base.yml

@@ -163,9 +163,9 @@ services:
       - MQTT_USER=__internal__
       - MQTT_CLIENT_ID=
       - MQTT_CLEAN_START=True
-      - MQTT_SUBSCRIBE_TOPIC=$$share/ingestors/PONTOS/+/+/+
+      - MQTT_SUBSCRIBE_TOPIC=$$share/ingestors/PONTOS_INGRESS/+/+/+
       - MQTT_SUBSCRIBE_TOPIC_QOS=0
-      - TOPIC_PARSER_FORMAT=PONTOS/{vessel_id:w}/{tag:w}/{index:d}
+      - TOPIC_PARSER_FORMAT=PONTOS_INGRESS/{vessel_id:w}/{tag:w}/{index:d}
       - PAYLOAD_MAP_FORMAT=timestamp=timestamp,value=value
       - PG_CONNECTION_STRING=postgres://pontos_user:${PONTOS_DB_PASSWORD}@db:5432/pontos
       - PG_TABLE_NAME=vessel_data.master
@@ -197,10 +197,7 @@ services:
       replicas: 3
     command:
       [
-        "mqtt --host emqx --port 1883 --user __internal__ subscribe -t '$$share/filterers/PONTOS/+/+/+' --line '{topic} {message}'
-        | sed -u '/^$/d'
-        | sed -u 's/PONTOS/PONTOS_HUB/'
-        | mqtt --host emqx --port 1883 --user __internal__ publish --line '{topic} {message}'"
+        "mqtt --host emqx --port 1883 --user __internal__ subscribe -t '$$share/filterers/PONTOS_INGRESS/+/+/+' --line '{topic} {message}' | sed -u '/^$/d' | sed -u 's/PONTOS_INGRESS/PONTOS_EGRESS/' | mqtt --host emqx --port 1883 --user __internal__ publish --line '{topic} {message}'"
       ]
 volumes:
   vol-emqx-data:

scripts/generate_publish_acl_rules.sh

@@ -2,7 +2,7 @@
 set -euo pipefail
 
 # Declarations
-topic_prefix='PONTOS'
+topic_prefix='PONTOS_INGRESS'
 topic_suffix=
 username=
 vessels=()
@@ -12,7 +12,7 @@ print_usage() {
     echo "Usage: script_name [-u username] [-t prefix] [-v vessel...] [-p parameter...]"
     echo "Options:"
     echo "  -u username        Set the username"
-    echo "  -t topic prefix    Set the topic prefix to use (default: PONTOS)"
+    echo "  -t topic prefix    Set the topic prefix to use (default: PONTOS_INGRESS)"
     echo "  -s topic suffix    Set the topic suffix to use (default: )"
     echo "  -v vessel          Specify a vessel (multiple occurrences allowed)"
     echo "  -p parameter       Specify a parameter (multiple occurrences allowed)"

tests/10-test-base-setup.bats

@@ -60,7 +60,7 @@ teardown_file() {
 
 @test "BASE: mqtt ingestion" {
     # Publish an actual payload that should be picked up by the ingestor and check that it gets written to the database
-    run docker run --network='host' hivemq/mqtt-cli:4.15.0 pub -v -h localhost -p 80 -ws -ws:path mqtt -t PONTOS/test_vessel/test_parameter/1 -m '{"timestamp": 12345678, "value": 42}'
+    run docker run --network='host' hivemq/mqtt-cli:4.15.0 pub -v -h localhost -p 80 -ws -ws:path mqtt -t PONTOS_INGRESS/test_vessel/test_parameter/1 -m '{"timestamp": 12345678, "value": 42}'
     assert_line --partial 'received PUBLISH acknowledgement'
 
     sleep 6
@@ -72,12 +72,12 @@ teardown_file() {
 
 @test "BASE: mqtt editor" {
     # Start a subscriber in the background and let it run for 10s
-    docker run --name subscriber --detach --network='host' hivemq/mqtt-cli:4.15.0 sub -v -h localhost -p 80 -ws -ws:path mqtt -t PONTOS_HUB/#
+    docker run --name subscriber --detach --network='host' hivemq/mqtt-cli:4.15.0 sub -v -h localhost -p 80 -ws -ws:path mqtt -t PONTOS_EGRESS/#
 
     sleep 2
 
     # Publish an actual payload that should be rewritten by the mqtt editor
-    run docker run --network='host' hivemq/mqtt-cli:4.15.0 pub -v -h localhost -p 80 -ws -ws:path mqtt -t PONTOS/test_vessel/test_parameter/1 -m '{"timestamp": 12345678, "value": 42}'
+    run docker run --network='host' hivemq/mqtt-cli:4.15.0 pub -v -h localhost -p 80 -ws -ws:path mqtt -t PONTOS_INGRESS/test_vessel/test_parameter/1 -m '{"timestamp": 12345678, "value": 42}'
     assert_line --partial 'received PUBLISH acknowledgement'
 
     sleep 1