1

Currently I have configured a solution with Fiware-Orion + Fiware-Cygnus + CKAN.

I have the following question:

Context attributes are always stored by type, value, and metadata; in row mode, as wel as in column mode. However, I am using more complex JSON structures in the Context Broker (type 'T')

However, most standard CKAN frontend viewers cannot handle JSON attribute values.

Is there any solution available to solve this?

A similar issue exists for the metadata, which are currently stored by Cygnus in CKAN as JSON by definition.

The simplest solution would be at change the context broker data model to use flat datatypes. However, I consider this a bad solution because I don't want the context modeling to suffer from storage adapter limitations.

Another solution would be to upgrade the standard viewers and write new ones. However, the following is a more generic solution that allows me to use any CKAN viewer.

Suppose I have a ContextElement attribute with name myAttr and value { a: 1, b: 2, c: 3}. Currently it would be stored by the CKAN Sink as:

Row persistence mode

attrName: "myAttr"
attrValue: "{ a: 1, b: 2, c: 3}"
attrMd: "[{name: md1_name, type: md1_type, value: md1_value}, {name: md2_name, type: md2_type, value: md2_value}]"

Column persistence mode:

column: myAttr
value:  { a: 1, b: 2, c: 3}
column: myAttr_md 
value [{name: md1_name, type: md1_type, value: md1_value}, {name: md2_name, type: md2_type, value: md2_value}]

To solve this issue I propose the following structure:

Row persistence mode:

attrName: myAttr_a
attrType: primitive myAttr.a type
attrValue: 1
attrName: myAttr_b
attrType: primitive myAttr.b type
attrValue: 2
attrName: myAttr_c
attrType: <primitive myAttr.b type>
attrValue: 3
attrName: myAttr_md_md1_name
attrType: md1_type
attrValue: md1_value
attrName: myAttr_md_md2_name
attrType: md2_type
attrValue: md2_value
...

Opinions appreciated.

Rian Wouters
  • 19
  • 2

1 Answers1

0

First of all, it must be said the way you are describing the "column"-like persistence is not correct. Having an attribute name myAttr, an attribute value { a: 1, b: 2, c: 3} and an attribute metadata [{name: md1_name, type: md1_type, value: md1_value}, {name: md2_name, type: md2_type, value: md2_value}], then only two columns are persisted:

  • A first column named myAttr with value { a: 1, b: 2, c: 3}
  • And a second column named myAttr_md with value [{name: md1_name, type: md1_type, value: md1_value}, {name: md2_name, type: md2_type, value: md2_value}]. I.e., all the metadata in a single column, not splited as you described.

You can find all the details regarding CKAN persistence at readthedocs.

Being said that, I think your use case could be easily fixed by changing your data model at Orion. Since it seems you need the single value { a: 1, b: 2, c: 3} to be splited per "sub-value", i.e. a: 1, b: 2 and c: 3 (the same applies to metadata), simply use 3 attributes with apropriate type and metadata.

frb
  • 3,738
  • 2
  • 21
  • 51
  • 1
    Though not at all essential for the discussion, I fixed the column persistence description. However, I of course I do not want to change my Orion data model just for the sake of having a decent historical dataset in CKAN. Anyway I can't do this for the metadata anyway since in either column mode and row mode, the metadata will be represented as a JSON structure in CKAN. There is currently no way of having the metadata values in a separate CKAN column. – Rian Wouters Sep 14 '16 at 12:28
  • I implemented my proposal including peristence tests and many other code improvements on https://github.com/rianwouters/fiware-cygnus/tree/feature/CKAN_structured_attributes – Rian Wouters Sep 14 '16 at 12:39
  • I'll check you code ;) Anyway, thinking a bit more on your needs, I think you are always assuming the type of the object fields regarding the notified attribute value are primitive ones. I'm saying this because of your proposal: `attrType: primitive myAttr.a type`... But you could be notified with `{"a":{"b":"1","c":"2"},"d":"3"}` (and very more complex things); in that case the type of `a` is not a primitive one, but a Json object, thus you could be in the same original scenario :) – frb Sep 14 '16 at 15:24
  • Of course. My implementation does not yet consider this, but can indeed be applied recursively. Have you seen examples of entities having this kind of complex json objects? – Rian Wouters Sep 15 '16 at 21:15