2

Good afternoon,

I'm trying to configure persistence of Orion Context's in the cosmos HDFS. So far I have created a Orion Context Broker Instance (orion-psb-image-R4.2), created a Cosmos user account and the folder 'cardio' inside it (/user/USERNAME/cardio).

I've also configured the cygnus.conf file (/usr/local/apache-flume-1.4.0-bin/conf/cygnus.conf) to be the following:

# APACHE_FLUME_HOME/conf/cygnus.conf
orionagent.sources = http-source
orionagent.sinks = hdfs-sink
orionagent.channels = notifications

# Flume source, must not be changed
orionagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
# channel name where to write the notification events
orionagent.sources.http-source.channels = notifications
# listening port the Flume source will use for receiving incoming notifications
orionagent.sources.http-source.port = 5050
# Flume handler that will parse the notifications, must not be changed
orionagent.sources.http-source.handler = es.tid.fiware.orionconnectors.cosmosinjector.OrionRestHandler
# regular expression for the orion version the notifications will have in their headers
orionagent.sources.http-source.handler.orion_version = 0\.11\.*
# URL target
orionagent.sources.http-source.handler.notification_target = /notify

# channel name from where to read notification events
orionagent.sinks.hdfs-sink.channel = notifications
# Flume sink that will process and persist in HDFS the notification events, must not be changed
orionagent.sinks.hdfs-sink.type = es.tid.fiware.orionconnectors.cosmosinjector.OrionHDFSSink
# IP address of the Cosmos deployment where the notification events will be persisted
orionagent.sinks.hdfs-sink.cosmos_host = 130.206.80.46
# port of the Cosmos service listening for persistence operations; 14000 for httpfs, 50070 for webhdfs and free choice for inifinty
orionagent.sinks.hdfs-sink.cosmos_port = 14000
# username allowed to write in HDFS (/user/myusername)
orionagent.sinks.hdfs-sink.cosmos_username = USERNAME
# dataset where to persist the data (/user/myusername/mydataset)
orionagent.sinks.hdfs-sink.cosmos_dataset = cardio
# HDFS backend type (webhdfs, httpfs or infinity)
orionagent.sinks.hdfs-sink.hdfs_api = httpfs

# channel name
orionagent.channels.notifications.type = memory
# capacity of the channel
orionagent.channels.notifications.capacity = 1000
# amount of bytes that can be sent per transaction
orionagent.channels.notifications.transactionCapacity = 100

The issue is that when I start the agent with the following command:

/usr/local/apache-flume-1.4.0-bin/bin/flume-ng agent --conf /usr/local/apache-flume-1.4.0-bin/conf/ -f /usr/local/apache-flume-1.4.0-bin/conf/cygnus.conf -n orionagent -Dflume.root.logger=INFO,consoleapache-flume-1.4.0-bin

I get the following error messages:

2015-07-17 15:33:10,477 (lifecycleSupervisor-1-1) [INFO - es.tid.fiware.orionconnectors.cosmosinjector.hdfs.HttpFSBackend.createDir(HttpFSBackend.java:68)] HttpFS operation: PUT http://130.206.80.46:14000/webhdfs/v1/user/arlourenco/cardio/?op=mkdirs&user.name=arlourenco HTTP/1.1
2015-07-17 15:33:10,708 (lifecycleSupervisor-1-1) [INFO - es.tid.fiware.orionconnectors.cosmosinjector.hdfs.HttpFSBackend.createDir(HttpFSBackend.java:71)] HttpFS response: HTTP/1.1 401 Unauthorized
2015-07-17 15:33:10,708 (lifecycleSupervisor-1-1) [ERROR - es.tid.fiware.orionconnectors.cosmosinjector.OrionHDFSSink.start(OrionHDFSSink.java:108)] The  directory could not be created in HDFS. HttpFS response: 401 Unauthorized
2015-07-17 15:33:17,470 (910628961@qtp-586740701-0) [WARN - org.apache.flume.source.http.HTTPSource$FlumeHTTPServlet.doPost(HTTPSource.java:186)] Received bad request from client.

And:

org.apache.flume.source.http.HTTPBadRequestException: orion/0.19.0 libcurl/7.19.7 user agent not supported

Here is the complete log:

+ exec /usr/lib/jvm/java-1.6.0-openjdk.x86_64/bin/java -Xmx20m -Dflume.root.logger=INFO,console -cp '/usr/local/apache-flume-1.4.0-bin/conf:/usr/local/apache-flume-1.4.0-bin/lib/*' -Djava.library.path= org.apache.flume.node.Application -f /usr/local/apache-flume-1.4.0-bin/conf/cygnus.conf -n orionagent
2015-07-17 15:33:09,858 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start(PollingPropertiesFileConfigurationProvider.java:61)] Configuration provider starting
2015-07-17 15:33:09,863 (conf-file-poller-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:133)] Reloading configuration file:/usr/local/apache-flume-1.4.0-bin/conf/cygnus.conf
2015-07-17 15:33:09,871 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:930)] Added sinks: hdfs-sink Agent: orionagent
2015-07-17 15:33:09,871 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:hdfs-sink
2015-07-17 15:33:09,872 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:hdfs-sink
2015-07-17 15:33:09,872 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:hdfs-sink
2015-07-17 15:33:09,872 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:hdfs-sink
2015-07-17 15:33:09,872 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:hdfs-sink
2015-07-17 15:33:09,872 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:hdfs-sink
2015-07-17 15:33:09,872 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:hdfs-sink
2015-07-17 15:33:09,873 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:hdfs-sink
2015-07-17 15:33:09,885 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:140)] Post-validation flume configuration contains configuration for agents: [orionagent]
2015-07-17 15:33:09,885 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:150)] Creating channels
2015-07-17 15:33:09,899 (conf-file-poller-0) [INFO - org.apache.flume.channel.DefaultChannelFactory.create(DefaultChannelFactory.java:40)] Creating instance of channel notifications type memory
2015-07-17 15:33:09,908 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:205)] Created channel notifications
2015-07-17 15:33:09,912 (conf-file-poller-0) [INFO - org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:39)] Creating instance of source http-source, type org.apache.flume.source.http.HTTPSource
2015-07-17 15:33:09,961 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:40)] Creating instance of sink: hdfs-sink, type: es.tid.fiware.orionconnectors.cosmosinjector.OrionHDFSSink
2015-07-17 15:33:09,967 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:119)] Channel notifications connected to [http-source, hdfs-sink]
2015-07-17 15:33:09,972 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:138)] Starting new configuration:{ sourceRunners:{http-source=EventDrivenSourceRunner: { source:org.apache.flume.source.http.HTTPSource{name:http-source,state:IDLE} }} sinkRunners:{hdfs-sink=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@e8234ec counterGroup:{ name:null counters:{} } }} channels:{notifications=org.apache.flume.channel.MemoryChannel{name: notifications}} }
2015-07-17 15:33:09,973 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:145)] Starting Channel notifications
2015-07-17 15:33:10,027 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:110)] Monitoried counter group for type: CHANNEL, name: notifications, registered successfully.
2015-07-17 15:33:10,028 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:94)] Component type: CHANNEL, name: notifications started
2015-07-17 15:33:10,028 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:173)] Starting Sink hdfs-sink
2015-07-17 15:33:10,029 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:184)] Starting Source http-source
2015-07-17 15:33:10,069 (lifecycleSupervisor-1-3) [INFO - org.mortbay.log.Slf4jLog.info(Slf4jLog.java:67)] Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2015-07-17 15:33:10,149 (lifecycleSupervisor-1-3) [INFO - org.mortbay.log.Slf4jLog.info(Slf4jLog.java:67)] jetty-6.1.26
2015-07-17 15:33:10,306 (lifecycleSupervisor-1-3) [INFO - org.mortbay.log.Slf4jLog.info(Slf4jLog.java:67)] Started SocketConnector@0.0.0.0:5050
2015-07-17 15:33:10,307 (lifecycleSupervisor-1-3) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:110)] Monitoried counter group for type: SOURCE, name: http-source, registered successfully.
2015-07-17 15:33:10,307 (lifecycleSupervisor-1-3) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:94)] Component type: SOURCE, name: http-source started
2015-07-17 15:33:10,456 (lifecycleSupervisor-1-1) [INFO - es.tid.fiware.orionconnectors.cosmosinjector.http.HttpClientFactory.<init>(HttpClientFactory.java:68)] Setting max total connections to 500 and default max connections per route to 100
2015-07-17 15:33:10,477 (lifecycleSupervisor-1-1) [INFO - es.tid.fiware.orionconnectors.cosmosinjector.hdfs.HttpFSBackend.createDir(HttpFSBackend.java:68)] HttpFS operation: PUT http://130.206.80.46:14000/webhdfs/v1/user/arlourenco/cardio/?op=mkdirs&user.name=arlourenco HTTP/1.1
2015-07-17 15:33:10,708 (lifecycleSupervisor-1-1) [INFO - es.tid.fiware.orionconnectors.cosmosinjector.hdfs.HttpFSBackend.createDir(HttpFSBackend.java:71)] HttpFS response: HTTP/1.1 401 Unauthorized
2015-07-17 15:33:10,708 (lifecycleSupervisor-1-1) [ERROR - es.tid.fiware.orionconnectors.cosmosinjector.OrionHDFSSink.start(OrionHDFSSink.java:108)] The  directory could not be created in HDFS. HttpFS response: 401 Unauthorized
2015-07-17 15:33:17,470 (910628961@qtp-586740701-0) [WARN - org.apache.flume.source.http.HTTPSource$FlumeHTTPServlet.doPost(HTTPSource.java:186)] Received bad request from client.
org.apache.flume.source.http.HTTPBadRequestException: orion/0.19.0 libcurl/7.19.7 user agent not supported
        at es.tid.fiware.orionconnectors.cosmosinjector.OrionRestHandler.getEvents(OrionRestHandler.java:93)
        at org.apache.flume.source.http.HTTPSource$FlumeHTTPServlet.doPost(HTTPSource.java:184)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:814)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:326)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
        at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

What could be the issue?

pschydlo
  • 23
  • 3

1 Answers1

0

First of all, you are using a pretty old version of Cygnus; I guess it is the one pre-installed in orion-psb-image-R4.2, which is 0.5.2. Latest one is 0.8.2, so please consider to upgrade, specially after enabling OAUth2 authentication/authorization in Cosmos' WebHDFS (you can find detailed instructions about upgrading here and detailed instructions about using OAuth2 with WebHDFS here).

Anyway, the problem is you are sending a user agent header different than orion/*, i.e. libcurl/7.19.7. Is it possible to avoid sending such a user agent from your client? If not, Cygnus code should be modified in order to accept all kind of user agents (something pointed by other users before, I admit it could be useful).

Community
  • 1
  • 1
frb
  • 3,738
  • 2
  • 21
  • 51
  • I've created the following issue https://github.com/telefonicaid/fiware-cygnus/issues/493. Let's try to avoid sending the `libcurl/*` user agent; if it not possible, then I'll implement the issue and release a 0.8.3 version of Cygnus. – frb Jul 17 '15 at 16:26
  • Thank you for the fast answer, I upgraded Cygnus to 0.8.2, set-up the oauth2 token and now it works perfectly. Quick question, how often do I need to refresh the access token? (In the response body it says the token is valid for 3600 [seconds?], is there any way to get a token that does not need to be constantly refreshed? – pschydlo Jul 20 '15 at 10:50
  • Great to hear it works :) The token should be renewed after the valid time given when requesting it... but current implementation of the Identity Manager we rely on (it is not my development) considers them valid forever (despite of such a message). We have created the following issue (https://github.com/telefonicaid/fiware-cygnus/issues/491) waiting for a change of behaviour. – frb Jul 20 '15 at 10:57
  • Thank you :-) Wish you a pleasant week! – pschydlo Jul 20 '15 at 15:02