1

I need to store data in cosmos with mysql format but I can't do it. I have checked that the data are stored on text file in cosmos, but when I enter in hive, there is not any table with my name.

The following is the config of cygnus. Is it correct? Where is my error to store in mysql tables?

cygnusagent.sources = http-source
cygnusagent.sinks = hdfs-sink mysql-sink
cygnusagent.channels = hdfs-channel mysql-channel

cygnusagent.sources.http-source.channels = hdfs-channel mysql-channel ckan-channel
cygnusagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
cygnusagent.sources.http-source.port = 5050
cygnusagent.sources.http-source.handler = es.tid.fiware.fiwareconnectors.cygnus.handlers.OrionRestHandler
cygnusagent.sources.http-source.handler.notification_target = /notify
cygnusagent.sources.http-source.handler.default_service = def_serv
cygnusagent.sources.http-source.handler.default_service_path = def_servpath
cygnusagent.sources.http-source.handler.events_ttl = 10
cygnusagent.sources.http-source.interceptors = ts de
cygnusagent.sources.http-source.interceptors.ts.type = timestamp
cygnusagent.sources.http-source.interceptors.de.type = es.tid.fiware.fiwareconnectors.cygnus.interceptors.DestinationExtractor$Builder
cygnusagent.sources.http-source.interceptors.de.matching_table = /usr/cygnus/conf/matching_table.conf

cygnusagent.sinks.hdfs-sink.channel = hdfs-channel
cygnusagent.sinks.hdfs-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink
cygnusagent.sinks.hdfs-sink.cosmos_host = 130.206.80.46
cygnusagent.sinks.hdfs-sink.cosmos_port = 14000
cygnusagent.sinks.hdfs-sink.cosmos_default_username = cristina.albaladejo
cygnusagent.sinks.hdfs-sink.cosmos_default_password = my_passw
cygnusagent.sinks.hdfs-sink.hdfs_api = httpfs
cygnusagent.sinks.hdfs-sink.attr_persistence = column
cygnusagent.sinks.hdfs-sink.hive_host = 130.206.80.46
cygnusagent.sinks.hdfs-sink.hive_port = 10000
cygnusagent.sinks.hdfs-sink.krb5_auth = false
cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_user = krb5_username
cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_password = xxxxxxxxxxxxx
cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_login_conf_file = /usr/cygnus/conf/krb5_login.conf
cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_conf_file = /usr/cygnus/conf/krb5.conf

# OrionMySQLSink configuration
cygnusagent.sinks.mysql-sink.channel = mysql-channel
cygnusagent.sinks.mysql-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionMySQLSink
cygnusagent.sinks.mysql-sink.mysql_host = 130.206.80.46
cygnusagent.sinks.mysql-sink.mysql_port = 3306
cygnusagent.sinks.mysql-sink.mysql_username = root
cygnusagent.sinks.mysql-sink.mysql_password = my_passw
cygnusagent.sinks.mysql-sink.attr_persistence = column

# hdfs-channel configuration
cygnusagent.channels.hdfs-channel.type = memory
cygnusagent.channels.hdfs-channel.capacity = 1000
cygnusagent.channels.hdfs-channel.transactionCapacity = 100

# mysql-channel configuration
cygnusagent.channels.mysql-channel.type = memory
cygnusagent.channels.mysql-channel.capacity = 1000
cygnusagent.channels.mysql-channel.transactionCapacity = 100
frb
  • 3,738
  • 2
  • 21
  • 51
Cristina
  • 161
  • 1
  • 1
  • 9
  • Could you please split the question into 2 different posts? I mean, you have 2 diffrent problems: Hive tables are not created, and Orion context data is not persisted in MySQL. Thanks! :) – frb May 07 '15 at 13:56
  • ok!! thanks. I am going to created another post. – Cristina May 08 '15 at 10:19

1 Answers1

0

As I answered in this Question:

How to configure Cygnus to save in mysql

The database tables are not created automatically in column mode. So you would have to create the tables.

The columns look like : recvTime - datetime , field1 , field2 .... field1_md - varchar , field2_md - varchar ....

If you change

cygnusagent.sinks.mysql-sink.attr_persistence = column

to

cygnusagent.sinks.mysql-sink.attr_persistence = row tables are created automatically, but I prefer the column way to save and handle data.

You can check cygnus log to see what is really happening. But seems to be that you have to create the database and the table with the fields declaration

You have also there my sample configuration wich is running OK

You can also search for cygnus proccess with ps aux | grep cygnus and kill it with kill XXXX(number of cygnus process)

And then run Cygnus in a terminal like this

/usr/cygnus/bin/cygnus-flume-ng agent --conf /usr/cygnus/conf/ -f /usr/cygnus/conf/YOURAGENTCONFIGFILE.conf -n cygnusagent -Dflume.root.logger=INFO,console

(You should change the name of your config file and your cygnusagent to run it)

I hope this helps you.

Community
  • 1
  • 1
Andrés Geezar
  • 108
  • 9