3

I'm running a custom Yarn Application using Apache Twill in HDP 2.5 cluster, but I'm not able to see my own container logs (syslog, stderr and stdout) when I go to my container web page: enter image description here

Also the login changes from my kerberos to "dr.who" when I navigate to this page. But I can see the logs of map-reduce jobs. Hadoop version is 2.7.3 and the cluster is yarn acl enabled.

insanely_sin
  • 986
  • 1
  • 14
  • 22
  • 1
    My 2 cents: the cluster is kerberized, YARN is kerberized, HistoryServer is kerberized... but the **YARN User Interface** is not kerberized -- anyone can connect to the web server without any authentication; the web server connects to HistoryServer with the hard-coded dummy name `Dr.Who`; so anyone can browse the jobs *(no ACL for that)*, but no-one can open the log files *(because of ACLs)*. – Samson Scharfrichter May 09 '17 at 21:02
  • 1
    Connect to the Edge Node with SSH, create a Kerberos ticket with `kinit`, then access the logs with command line `yarn logs -applicationId application_0000000_0000` -- that command line *is* kerberized because it uses low-level RPC calls. And don't forget `| more` because you will get an awful lot of content. – Samson Scharfrichter May 09 '17 at 21:05
  • Hi @SamsonScharfrichter I tried that actually but when the app is live and running I get the following message after running yarn logs command: **"can not find any log file matching the pattern: [all] for the container: ..."**, but when I kill the application, I can see all the logs. I checked that the log files are not present actually in this directory "/yarn/logs//logs//" when the app is running, it gets created only when I kill the application. Do I need to set some Yarn log configs manually so that I can see the logs real time using this command? – insanely_sin May 10 '17 at 04:01
  • 1
    _"set configs ... [to] see the logs real time"_ > You can't. Logs are collected by HistoryServer when the job AM terminates the execution. That's one of the pains of long-running streaming apps (and services) which has no good solution yet, cf. https://www.slideshare.net/HadoopSummit/running-services-on-yarn/23 – Samson Scharfrichter May 10 '17 at 07:40
  • 1
    Right now, if you want to "stream" logs out of a running container, you have to build your own streaming system -- e.g. start a Flume agent on the Edge Node; in your app, define a specific Log4J output for "applicative" logs, using the Flume plug-in, to stream these logs to the agent; set up the agent to flush results to a local file every N seconds. – Samson Scharfrichter May 10 '17 at 07:47
  • 1
    FYI, twill will stream the application logs back to the client via a Kafka queue that it sets up. – Martin Serrano May 10 '17 at 16:52
  • Hi @MartinSerrano ! I know that logs will be sent back to the client but it was really very convenient to see the stderr and stdout from the container node web UI itself. I was able to do that in HDP 2.2 but no longer in HDP 2.5 – insanely_sin May 10 '17 at 17:17
  • Hi @SamsonScharfrichter :) you've been very helpful, I just need to know one thing that I don't really need to stream the logs out of the container, but if I'm able to see the stdout of a container that is sufficient for me. I'm able to see the syslogs, stderrs etc for map-reduce jobs in the same cluster using the same web UI and earlier in HDP 2.2 I was able to see stdouts of Yarn application as well. Can I specify [ApplicationAccessType](http://hadoop.apache.org/docs/r2.7.1/api/org/apache/hadoop/yarn/api/records/ApplicationAccessType.html) from any yarn config property for a yarn app? – insanely_sin May 10 '17 at 18:21
  • No idea... But again, AFAIK, `stdout` is captured by the YARN container wrapper, and shipped to HistoryServer just before the container resources are released. – Samson Scharfrichter May 10 '17 at 18:59

1 Answers1

2

i had this issue with hadoop ui. I found in this doc, that the hadoop.http.staticuser.user is set to dr.who by default and you need include it in the related setting file (in my issue is core-site.xml file). so late but hope useful.

Hossein Vatani
  • 1,381
  • 14
  • 26