How to process sensor time series data in batch (e.g. every 1 minute) to time series database (e.g. CrateDB) via FIWARE ORION CONTEXT BROKER?

Question

I want to process sensor data into a time series database via FIWARE ORION CONTEXT BROKER. However, instead of writing each data individually, I want to write it collectively as a 1-minute time series. Is something like this possible?

Could you help me? What path should I follow?

Answer 1

You need to split this problem into separate parts. Firstly define the payload of your batch operation. If the device can collect a series of readings and then deliver them as NGSI you can use the /v2/op/update/ for NGSI-v2 or /ngsi-ld/v1/entityOperations/upsert for NGSI-LD

For NGSI-v2 it would be something like

curl -L -X POST 'http://localhost:1026/v2/op/update/' \
-H 'Content-Type: application/json' \
--data-raw '{
  "actionType":"append",
  "entities":[
    {
      "id": "urn:ngsi-ld:TemperatureSensor:002",
      "type": "TemperatureSensor",
      "temperature": {
            "type": "Property",
            "value": 21,
            "metadata":{
                "unitCode": "CEL",
                "TimeInstant": {
                    "type": "DateTime",
                    "value": "XXXX-XXX-XXXXX"
                 }
           }
      }
    },
    {
      "id": "urn:ngsi-ld:TemperatureSensor:003",
      "type": "TemperatureSensor",
      "temperature": {
            "type": "Property",
            "value": 27,
            "metadata":{
                "unitCode": "CEL",
                "TimeInstant": {
                    "type": "DateTime",
                    "value": "XXXX-XXX-XXXXX"
                 }
           }
      }
    }
  ]
}'

For NGSI-LD it would be something like this:

curl -L -X POST 'http://localhost:1026/ngsi-ld/v1/entityOperations/upsert' \
-H 'Content-Type: application/json' \
-H 'Link: <http://context/ngsi-context.jsonld>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"' \
--data-raw '[
    {
      "id": "urn:ngsi-ld:TemperatureSensor:002",
      "type": "TemperatureSensor",
      "temperature": {
            "type": "Property",
            "value": 21,
            "unitCode": "CEL",
            "observedAt": "XXXXX-XXXX-XXXXXX"
      }
    },
    {
      "id": "urn:ngsi-ld:TemperatureSensor:003",
      "type": "TemperatureSensor",
      "temperature": {
            "type": "Property",
            "value": 27,
            "unitCode": "CEL",
            "observedAt": "XXXXX-XXXX-XXXXXX"
      }
    }
]'

The important thing here is to include the observedAt or TimeInstant in the payload so that it can be passed when creating a subscription into QuantumLeap (amend the fiware-service to the correct Tenant:

curl -iX POST \
  'http://localhost:1026/v2/subscriptions/' \
  -H 'Content-Type: application/json' \
  -H 'fiware-service: openiot' \
  -H 'fiware-servicepath: /' \
  -d '{
  "description": "Notify QuantumLeap of temperature changes",
  "subject": {
    "entities": [
      {
        "idPattern": "TemperatureSensor.*"
      }
    ],
    "condition": {
      "attrs": [
        "count"
      ]
    }
  },
  "notification": {
    "http": {
      "url": "http://quantumleap:8668/v2/notify"
    },
    "attrs": [
      "temperature"
    ],
    "metadata": ["dateCreated", "TimeInstant"]
  },
  "throttling": 1
}'

Now it could be the case that you have a device writing to a file and periodically uploading a batch of readings. In this case you can create a Custom IoT Agent which deciphers the payload and transforms each reading into a batch upsert request - an example can be found in this FIWARE Video on YouTube.

The steps are always the same:

1. Ingest the payload holding the readings (in format X)
2. Transform format X to an in-memory representation.
3. Tweak the representation into the entities part of an NGSI request
4. Send the upsert to the context broker.

The subscription to QuantumLeap will persist the data to a database where you can make time-series queries. The FIWARE Tutorial will cover the steps.

Now if you are using NGSI-LD you can still use QuantumLeap, but you could also look at using the Temporal interface directly. More details in the video here