Can't use attr_persistence column on mysql sink

Question

I'm trying to use the column type with OrionMySQLSink. My agent has this code:

cygnusagent.sinks.mysql-sink.type = com.telefonica.iot.cygnus.sinks.OrionMySQLSink
cygnusagent.sinks.mysql-sink.channel = mysql-channel
cygnusagent.sinks.mysql-sink.enable_grouping = false
cygnusagent.sinks.mysql-sink.mysql_host = localhost
cygnusagent.sinks.mysql-sink.mysql_port = 3306
cygnusagent.sinks.mysql-sink.mysql_username = ********
cygnusagent.sinks.mysql-sink.mysql_password = *********
cygnusagent.sinks.mysql-sink.table_type = table-by-destination
cygnusagent.sinks.mysql-sink.attr_persistence = column
cygnusagent.sinks.mysql-sink.batch_size = 1
cygnusagent.sinks.mysql-sink.batch_timeout = 10

I'm having this error when I use the sink:

2015-12-14 08:43:58,118 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - com.telefonica.iot.cygnus.sinks.OrionSink.process(OrionSink.java:187)] Persistence error (Unknown database 'fiw-serv')

I don't have any problem with row mode, only with column mode.

score 0 · Accepted Answer · edited Jun 20 '20 at 09:12

That's because the column persistence mode of OrionMySQLSink requires the table is created in advance. A detailed explanation can be found here. Basically:

Please observe not always the same number of attributes is notified; this depends on the subscription made to the NGSI-like sender. This is not a problem for the row persistence mode, since fixed 8-fields data rows are inserted for each notified attribute. Nevertheless, the column mode may be affected by several data rows of different lengths (in term of fields). Thus, the column mode is only recommended if your subscription is designed for always sending the same attributes, event if they were not updated since the last notification.

In addition, when running in column mode, due to the number of notified attributes (and therefore the number of fields to be written within the Datastore) is unknown by Cygnus, the table can not be automatically created, and must be provisioned previously to the Cygnus execution. That's not the case of the row mode since the number of fields to be written is always constant, independently of the number of notified attributes.

Something similar occurs with OrionCKANSink, where the resource, datastore and view must be created in advance.

Can't use attr_persistence column on mysql sink

1 Answers1