How to store hierarchical data through FIWARE Cygnus with MongoDB

Question

I've deployed a FIWARE configuration to receive MQTT messages from TheThingsNetwork. The configuration uses IoTAgent-JSON, Orion and Cygnus, with a MongoDB backend for Cygnus.

The messages are correctly persisted to MongoDB. Unfortunately, the message data having a hierarchical structure, some parts of the message appear as strings in MongoDB, instead of embedded subdocuments.

This makes it difficult to query the data.

Here are the details:

The message format is defined by the TTN MQTT Data API.

I've defined an entity type on Orion as follows:

curl http://localhost:1026/v2/entities -X POST -H "content-type: application/json" -H "fiware-service: myservice" -H "fiware-servicepath: /mypath" -d @- << EOF
{
    "id": "TtnMqttMessage",
    "type": "TtnMqttMessge",
    "app_id": { "type": "Text", "value": "my-app-id" },
    "dev_id": { "type": "Text", "value": "my-dev-id" },
    ...
    "metadata": { "type": "StructuredValue", "value": {
        "airtime": 46336000,
        "time": "1970-01-01T00:00:00Z",
        ...
        "gateways": 
        [ 
            {
                "gw_id": "gw1",
                "timestamp": 12345,
                "time": "1970-01-01T00:00:00Z",
                ...
                "altitude": 6
            }
        ]
    } }
}
EOF

In particular, in the configuration above, metadata is a structured value, containing an array of gateways.

On IoTAgent-JSON, a service and a device are provisioned:

curl http://localhost:4041/iot/services -X POST -H "content-type: application/json" -H "fiware-service: myservice" -H "fiware-servicepath: /mypath" -d @- << EOF
{
    "services": [
        {
            "apikey": "my_app_id",
            "entity_type": "TtnMqttMessage",
            "resource": "/iot/json"
        }
    ]
}
EOF

curl http://localhost:4041/iot/devices?options=keyValues -X POST -H "content-type: application/json" -H "fiware-service: myservice" -H "fiware-servicepath: /mypath" -d @- << EOF
{
    "devices": [{
        "device_id": "my_device_id",
        "entity_name": "TtnMqttMessage",
        "entity_type": "TtnMqttMessage",
        "timezone": "Europe/Zurich",
        "transport": "MQTT"
    }]
}
EOF

Finally, a notification subscription is established from Orion to Cygnus:

curl http://localhost:1026/v1/subscribeContext -H "content-type: application/json" -H "fiware-service: myservice" -H "fiware-servicepath: /mypath" -X POST  -d @- << EOF
{
    "entities": [
        {
            "type": "TtnMqttMessage",
            "isPattern": "false",
            "id": "TtnMqttMessage"
        }
    ],
    "attributes": [ "app_id", "dev_id", "hardware_serial", "port", "counter",  "is_retry",  "confirmed", "payload_raw", "payload_fields", "metadata" ],
    "reference": "http://cygnus:5050/notify",
    "duration": "P100Y",
    "notifyConditions": [
        {
            "type": "ONCHANGE",
            "condValues": [ "app_id", "dev_id", "payload_raw", "counter" ]
        }
    ]
}
EOF

The received messages are persisted to MongoDB:

> mongo
> ...
> db['..collectionname...'].findOne();
{
    "_id" : ObjectId("5adf0b904cedfd001cd72113"),
    "recvTime" : ISODate("2018-04-24T10:48:47.605Z"),
    "app_id" : "my-app-id",
    "confirmed" : "false",
    "counter" : "2",
    "dev_id" : "my-dev-id",
    "hardware_serial" : "0102030405060708",
    "is_retry" : "false",
    "metadata" : "{\"airtime\":4.6336e+07,\"time\":\"1970-01-01T00:00:00Z\",\"frequency\":868.1,\"modulation\":\"LORA\",\"data_rate\":\"SF7BW125\",\"bit_rate\":50000,\"coding_rate\":\"4/5\",\"latitude\":52.2345,\"longitude\":6.2345,\"altitude\":2,\"gateways\":[{\"gw_id\":\"gw1\",\"timestamp\":12345,\"time\":\"1970-01-01T00:00:00Z\",\"channel\":0,\"rssi\":-25,\"snr\":5,\"rf_chain\":0,\"latitude\":52.1234,\"longitude\":6.1234,\"altitude\":6}]}",
    "payload_fields" : "{}",
    "payload_raw" : "AQIDBA",
    "port" : "1"
}

As can be seen above, the attribute metadata, and in particular the array gateways it contains, are stored as strings and not as JSON subdocuments.

How can I persist the data in a format that can be easily queried? Eg.

with an embedded subdocument under metadata (denormalized form)
or in normalized form where metadata is a separate document that references the main document for the message.

score 0 · Accepted Answer · answered May 11 '18 at 11:41

0

The created entity looks well structured, I notice that you create the entity using the v2 but you subscribe to receive the notifications using v1, I know that is a correct way to do that but maybe the wrong behavior is generated by that reason.

answered May 11 '18 at 11:41

anmunoz

131
6

Dear anmunoz, thank's for the suggestion. I've created a subscription using v2. This requires using `"attrsFormat": "legacy"`, since Cygnus only implements NGSIv1, as far as I know. The resulting data in MongoDB is unfortunately the same as before. I also tried `normalized`, `keyValues` and `values` as attrsFormat. However, this resulted in errors during the subscription. – Juergen May 15 '18 at 10:28
Hi Juergen, I have been testing and you are right, using v2 subscriptions doesn't solve the problem. As you mention, Cygnus only implements NGSIv1 that course the issue for storing your metadata field instead as a JSON object. Currently, we are working on the update of Cygnus fro being able to manage NGSIv2 in a native way but it still is a work in progress. As soon as we have the new version available I will tell you. – anmunoz May 18 '18 at 08:48
Thanks a lot for all your effort. For the moment I wrote a script to parse and handle the JSON text outside FIWARE. Hopefully it won't be necessary anymore when the Cygnus update comes out. – Juergen May 21 '18 at 19:54

How to store hierarchical data through FIWARE Cygnus with MongoDB

1 Answers1