2

We manage to integrate idas-cygnus-orion flow, everything is working perfectly. (Thanks for your help Francisco)

Now we want to add the Cosmos to the integration and I've configured Cygnus with your answer posted here (Part starting with * is added for Cosmos configuration)

When I start Cygnus, we get this error msg

        [root@idas-new centos]# tail -500f nohup.out
Warning: No configuration directory set! Use --conf <dir> to override.
Warning: JAVA_HOME is not set!
+ exec /usr/bin/java -Xmx20m -Dflume.root.logger=DEBUG,console -cp '/usr/cygnus/lib/*:/usr/cygnus/plugins.d/cygnus/lib/*:/usr/cygnus/agent_test.conf -n cygnusagent
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/cygnus/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/cygnus/plugins.d/cygnus/lib/cygnus-0.12.0_SNAPSHOT-jar-with-dependencies.jar!/org/slf4j/impl/
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
16/05/18 09:26:46 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting
16/05/18 09:26:46 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:/usr/cygnus/conf/agent_test.con
16/05/18 09:26:46 INFO conf.FlumeConfiguration: Processing:hdfs-sink
16/05/18 09:26:46 INFO conf.FlumeConfiguration: Processing:hdfs-sink
16/05/18 09:26:46 INFO conf.FlumeConfiguration: Processing:hdfs-sink
16/05/18 09:26:46 INFO conf.FlumeConfiguration: Processing:hdfs-sink
16/05/18 09:26:46 INFO conf.FlumeConfiguration: Processing:hdfs-sink
16/05/18 09:26:46 INFO conf.FlumeConfiguration: Processing:hdfs-sink
16/05/18 09:26:46 INFO conf.FlumeConfiguration: Processing:hdfs-sink
16/05/18 09:26:46 INFO conf.FlumeConfiguration: Processing:hdfs-sink
16/05/18 09:26:46 INFO conf.FlumeConfiguration: Added sinks: hdfs-sink Agent: cygnusagent
16/05/18 09:26:46 INFO conf.FlumeConfiguration: Processing:hdfs-sink
16/05/18 09:26:46 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [cygnusagent]
16/05/18 09:26:46 INFO node.AbstractConfigurationProvider: Creating channels
16/05/18 09:26:46 INFO channel.DefaultChannelFactory: Creating instance of channel hdfs-channel type memory
16/05/18 09:26:46 INFO node.AbstractConfigurationProvider: Created channel hdfs-channel
16/05/18 09:26:46 INFO source.DefaultSourceFactory: Creating instance of source http-source, type org.apache.flume.source.http.HTTPS
16/05/18 09:26:46 INFO handlers.OrionRestHandler: Cygnus version (0.12.0_SNAPSHOT.d6ee0e4548ca6bf9116c1dee4c4cbcc7da8b1752)
16/05/18 09:26:46 INFO handlers.OrionRestHandler: Startup completed
16/05/18 09:26:46 INFO sink.DefaultSinkFactory: Creating instance of sink: hdfs-sink, type: com.telefonica.iot.cygnus.sinks.OrionHDF
16/05/18 09:26:46 ERROR node.AbstractConfigurationProvider: Sink hdfs-sink has been removed due to an error during configuration
java.lang.NullPointerException
        at com.telefonica.iot.cygnus.sinks.OrionHDFSSink.configure(OrionHDFSSink.java:339)
        at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
        at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:418)
        at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:103)
        at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurati
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
        at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
16/05/18 09:26:46 INFO node.AbstractConfigurationProvider: Channel hdfs-channel connected to [http-source]
16/05/18 09:26:46 INFO node.Application: Starting new configuration:{ sourceRunners:{http-source=EventDrivenSourceRunner: { source:oyChannel{name: hdfs-channel}} }
16/05/18 09:26:46 INFO node.Application: Starting Channel hdfs-channel
16/05/18 09:26:46 INFO instrumentation.MonitoredCounterGroup: Monitoried counter group for type: CHANNEL, name: hdfs-channel, regist
16/05/18 09:26:46 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: hdfs-channel started
16/05/18 09:26:46 INFO node.Application: Starting Source http-source
16/05/18 09:26:46 INFO interceptors.GroupingRules: No grouping rules have been read
Exception in thread "Thread-1" java.lang.NullPointerException
        at java.io.File.<init>(Unknown Source)
        at com.telefonica.iot.cygnus.interceptors.GroupingInterceptor$ConfigurationReader.run(GroupingInterceptor.java:244)
16/05/18 09:26:46 INFO mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
16/05/18 09:26:46 INFO mortbay.log: jetty-6.1.26
16/05/18 09:26:47 INFO mortbay.log: Started SocketConnector@0.0.0.0:5050

We have this grouping_rules.conf file in our path, but inside, everything is commented(since we didnt need this file for the CKAN integration) do we need to edit this file for Cosmos?(if yes, how?)

Would you please help us on this issue?

Many thanks Omer

cygnusagent.sources = http-source
cygnusagent.sinks = ckan-sink hdfs-sink
cygnusagent.channels = ckan-channel hdfs-channel

cygnusagent.sources.http-source.channels = ckan-channel hdfs-channel
cygnusagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
cygnusagent.sources.http-source.port = 5050
cygnusagent.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.OrionRestHandler
cygnusagent.sources.http-source.handler.notification_target = /notify
cygnusagent.sources.http-source.handler.events_ttl = 2
cygnusagent.sources.http-source.interceptors = ts gi
cygnusagent.sources.http-source.interceptors.ts.type = timestamp
cygnusagent.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.GroupingInterceptor$Builder
cygnusagent.sources.http-source.interceptors.gi.gropuing_rules_conf_file = /etc/cygnus/conf/grouping_rules.conf

cygnusagent.channels.ckan-channel.type = memory
cygnusagent.channels.ckan-channel.capacity = 1000
cygnusagent.channels.ckan-channel.transactionCapacity = 100

cygnusagent.sinks.ckan-sink.channel = ckan-channel
cygnusagent.sinks.ckan-sink.type = com.telefonica.iot.cygnus.sinks.OrionCKANSink
cygnusagent.sinks.ckan-sink.enable_grouping = false
cygnusagent.sinks.ckan-sink.api_key = XXXXXXXXXX
cygnusagent.sinks.ckan-sink.ckan_host = data.lab.fiware.org
cygnusagent.sinks.ckan-sink.ckan_port = 443
cygnusagent.sinks.ckan-sink.orion_url = localhost:1026
cygnusagent.sinks.ckan-sink.attr_persistence = row
cygnusagent.sinks.ckan-sink.ssl = true
cygnusagent.sinks.ckan-sink.batch_size = 1
cygnusagent.sinks.ckan-sink.batch_timeout = 10

cygnusagent.sinks.hdfs-sink.type = com.telefonica.iot.cygnus.sinks.OrionHDFSSink
cygnusagent.sinks.hdfs-sink.channel = hdfs-channel
cygnusagent.sinks.hdfs-sink.enable_grouping = false
cygnusagent.sinks.hdfs-sink.hdfs_host = cosmos.lab.fiware.org
cygnusagent.sinks.hdfs-sink.hdfs_port = 14000
cygnusagent.sinks.hdfs-sink.hdfs_username = XXXXX
cygnusagent.sinks.hdfs-sink.hdfs_password = XXX
cygnusagent.sinks.hdfs-sink.oauth2_token = XXXXXXXXX
cygnusagent.sinks.hdfs-sink.file_format = json-column


cygnusagent.channels.hdfs-channel.type = memory
cygnusagent.channels.hdfs-channel.capacity = 1000
cygnusagent.channels.hdfs-channel.transactionCapacity = 100
Community
  • 1
  • 1
dmtzz
  • 223
  • 2
  • 8

1 Answers1

2

Everything seems to be OK with the configuration except for:

cygnusagent.sources.http-source.interceptors.gi.gropuing_rules_conf_file = /etc/cygnus/conf/grouping_rules.
conf
  • gropuing -> grouping (this probably comes from the Quick Start Guide, where this typo was living until we fixed).
  • conf part of the file name seems to be in a second line... Flume configurations don't like this, it is better all about a parameter is within the same line.

Try this changes and let see.

frb
  • 3,738
  • 2
  • 21
  • 51
  • thank you Francisco, I've edited this but as I said above, the content of this file is empty, since I couldn't find a "how to use guide" for the grouing rules conf file, as the content if this file all commented, we get this error now: `16/05/18 09:55:04 INFO interceptors.GroupingRules: Grouping rules read: 16/05/18 09:55:04 ERROR interceptors.GroupingRules: Error while parsing the Json-based grouping rules file. Details=null 16/05/18 09:55:04 WARN interceptors.GroupingRules: Grouping rules syntax has errors` how can i add a rule into this file and how does it work? – dmtzz May 18 '16 at 09:56
  • You can check this link: http://fiware-cygnus.readthedocs.io/en/latest/cygnus-ngsi/installation_and_administration_guide/grouping_rules_conf/index.html. Nevertheless, if you don't need the grouping rules, just create the file (it must exist) and add this content: `{"grouping_rules":[]}`. – frb May 18 '16 at 10:13
  • added empty grouping rule as you posted above, its ok but the hdfs sink gives null pointer exception while start: `16/05/18 10:17:52 ERROR node.AbstractConfigurationProvider: Sink hdfs-sink has been removed due to an error during configuration java.lang.NullPointerException at com.telefonica.iot.cygnus.sinks.OrionHDFSSink.configure(OrionHDFSSink.java:339)` – dmtzz May 18 '16 at 10:20
  • Yes, they are differente problems. Once fixed the grouping rules, let's see the sink. Which version are you running? How did you install Cygnus? Can you share the entire log before the null pointer exception? Thanks. – frb May 18 '16 at 13:28
  • hello frb, yep the grouping rules has been fixed, im posting the full log in my original question above. well you've sent me the snapshot version,(i've found out a bug and, you guys fixed this and sent me the snapshot version) – dmtzz May 18 '16 at 13:52
  • added the entire log in my post/question above – dmtzz May 18 '16 at 13:53
  • im starting cygnus like this(as you suggested me like 2 months ago): `nohup /usr/cygnus/bin/cygnus-flume-ng agent —conf /usr/cygnus/conf -f /usr/cygnus/conf/agent_test.conf -n cygnusagent -Dflume.root.logger=DEBUG,console &` – dmtzz May 18 '16 at 13:56
  • You are using 0.12.0 version of Cygnus... OK, line 339 at OrionHDFSSink is about the configuration parameter `hive.server_version`, which only has sense if `hive` is configured. Could you try adding this line to your HDFS sink configuration: `cygnusagent.sinks.hdfs-sink.hive = false`? – frb May 18 '16 at 16:41
  • i have added this line, this time cygnus gets exception at line 447 :( `16/05/19 06:23:24 ERROR node.AbstractConfigurationProvider: Sink hdfs-sink has been removed due to an error during configuration java.lang.NullPointerException at com.telefonica.iot.cygnus.sinks.OrionHDFSSink.configure(OrionHDFSSink.java:447) ` – dmtzz May 19 '16 at 06:25
  • This is quite rare because line 447 is out of the `configure` method in `OrionHDFSSink.java` at version 0.12.0 (please, check https://github.com/telefonicaid/fiware-cygnus/blob/release/0.12.0/src/main/java/com/telefonica/iot/cygnus/sinks/OrionHDFSSink.java#L447). It seems as you are really running 0.13.0... nevertheless the logs say you are running 0.12.0. I would recommend you to remove Cygnus from that machine and start from the scratch by doing `yum install cygnus` and installing 0.13.0. – frb May 23 '16 at 13:53
  • hmm strange, is there any common way to remove the current one and install it with `yum`? I was using the `cygnus-common--jar-with-dependencies.jar` snapshot that you built and sent for me – dmtzz May 23 '16 at 14:26
  • Ummm, I did not sent you anything if I remember well :) Anyway, until 0.13.0 (included), Cygnus was built as a single jar. From 1.0.0 (included), Cygnus was split into cygnus-common and cygnus-ngsi (which relies on cygnus-common). Using only cygnus-common will not do anything apart from running a "plain" Apache Flume (continues). – frb May 23 '16 at 15:03
  • (continuing) Being said that, my recommendation is you install 0.13.0, which is installed by doing `yum install cygnus` (you'll need the FIWARE repo installed in your machine). In order to remove the current one, I'll do the following: `sudo rpm -e -vv --allmatches --nodeps --noscripts --notriggers cygnus`. Additionally, I'll check you do't jave any "cygnus"-named folder in you machine. – frb May 23 '16 at 15:03
  • well you did send me :D please check this long converstaion in this ticket HELP-5978. I ended up editing the config file and using the snapshot jar you've sent to me since i found there was a [bug])(https://github.com/telefonicaid/fiware-cygnus/issues/826) in cygnus. so should I delete the existing one with the command above and install 0.13 and configure it again from scratch? does this remove command works for the current one :D ? – dmtzz May 23 '16 at 15:10
  • You can save the configuration somewhere else before removing Cygnus :) And you are right, I've checked it and I sent you the 0.12.0 jar, sorry for the confussion. Nevertheless, try installing 0.13.0, please, it is much more stable. – frb May 23 '16 at 15:19
  • i did the following `sudo rpm -e -vv --allmatches --nodeps --noscripts --notriggers cygnus` but it seems that /usr/cygnus folder and `./conf` and `./plugins.d` directory and files under them do still exist, should I continue installing `yum install cygnus` or do I need to do something else to delete these folders or another ones? – dmtzz May 23 '16 at 15:39
  • I think the files under `/usr/cygnus/conf/` can be left. Nevertheless, please check `/usr/cygnus/plugins.d/lib` is empty (if not, just remove its content, but the folder must exist). – frb May 23 '16 at 15:59
  • upgraded to version 0.13 :) have some questions about hdfs sink/channel, but before that there is an issue with the port. the management port(which I configured to run on 8085 with version 0.12) suddenly changed to 8081(which mongodb is running on this). why does cygnus v 0.13 ignore the conf in cygnus_instance_1.conf? which I have like this: `ADMIN_PORT=8085` – dmtzz May 23 '16 at 16:49
  • How are you running this new instance? As a service (i.e. `service cygnus start`)? Or as a process (i.e. `nohup /usr/cygnus/bin/cygnus-flume-ng agent ...`)? In the first case, which are the names of the cygnus_instance and agent configuration files? In the second case, which is the exact command you are using? – frb May 24 '16 at 05:28
  • good morning frb, I'm using the 2nd approach for version 0,13. `nohup /usr/cygnus/bin/cygnus-flume-ng agent —conf /usr/cygnus/conf -f /usr/cygnus/conf/agent_test.conf -n cygnusagent -Dflume.root.logger=DEBUG,console &` – dmtzz May 24 '16 at 06:47
  • Good morning! :) That makes sense. When running as a process, the `cygnus_instance_.conf` file is not used at all, only the `agent_.conf` file is read. In that case, in order to change the default admin port you must use the `-p` option: `nohup /usr/cygnus/bin/cygnus-flume-ng agent —conf /usr/cygnus/conf -f /usr/cygnus/conf/agent_test.conf -n cygnusagent -Dflume.root.logger=DEBUG,console -p 8085 &` – frb May 24 '16 at 09:21
  • thanks, I've started with this optionm now I have a last question(i hope :D ) please check the startup logs of cygnus pasted here: `http://pastebin.com/gNwQAz7n` i see the ckan sink and ckan channel is started as in the logs, but for the hdfs, I see these 2 lines: `Added sinks: ckan-sink hdfs-sink Agent: cygnusagent` and `no context for sinkhdfs-sink` do you think that hdfs sink/channel is started sucessfully? Cygnus doesnt persist the received data to hdfs sik, but only to Ckan sink :( – dmtzz May 24 '16 at 10:27
  • I don't think so :( This log `16/05/24 10:18:52 WARN conf.FlumeConfiguration: no context for sinkhdfs-sink` means something is wrong with the HDFS sink configuration. I've tried to reproduce the error using your configuration (even, using the fake `xxxxx` values) in a fresh 0.13.0 installation and the log is different: http://pastebin.com/fYVb6nMS As you can see, both sinks seem to start properly. Are you sure nothing has been added to the configuration you originally posted apart for the Hive parameter I suggested? Can you update it, please? – frb May 24 '16 at 13:06
  • hello Francisco, it seems my configuration is the same with yours :( please have a look at it `http://pastebin.com/PEpAf2nw` – dmtzz May 24 '16 at 13:54
  • oh god :D updated and everything is working fine, thanks so much Francisco! – dmtzz May 24 '16 at 14:57