0

In my Apache mod_jk log I have entries in the format like:

[Wed Oct 26 10:59:43 2011] [3732:2460] [error] jk_ajp_common.c (1618): (myJbossServer) Tomcat is down or network problems. Part of the response has already been sent to the client

RedHat's Troubleshooting Guide says,

Note that with these messages the [11159:3086420192] portion of the message serves as an identifier for the connection/request in question. Therefore tracing back from the point of the error in logs can help clarify the activity around the connection/request that lead to the error.

How do I do this? I cannot find the identifier in my JBoss logs. I don't see any identifiers in the JBoss logs, errors, yes, tracking numbers, no. Where do I need to be looking to track this identifier down? I'm not really looking for a solution to the problem, only how to use the identifier and check between Apache mod_jk and Jboss.

EDIT: I am asking where can I find the identifier [3732:2460] in the myJBossServer logs.

johnny
  • 2,328
  • 9
  • 37
  • 57

1 Answers1

1

Actually it's in front of you, the failed worker is the one inside brackets, called myJbossServer.

Check you workers.properties for that worker and you will find it's IP or hostname.

Around that time hopefully you'll get hints from your JBoss / operating system (sar get's very handy if you were experiencing problems with CPU / Memory / ecc BTW).

Actualy you made me curious about the numbers between square brackets. I tried searching the documentation but wasn't able to find what it was. Checking the source code though brought this:

int jk_log(jk_logger_t *l,
           const char *file, int line, const char *funcname, int level,
           const char *fmt, ...)
{

...

        if (line) { /* line==0 only used for request log item */
            /* Log [pid:threadid] for all levels except REQUEST.
             * This information helps to correlate lines from different logs.
             * Performance is no issue, because with production log levels
             * we only call it often, if we have a lot of errors
             */
            rc = snprintf(buf + used, usable_size - used,
                          "[%" JK_PID_T_FMT ":%" JK_PTHREAD_T_FMT "] ", getpid(), jk_gettid());

Looking at the RedHat document you pasted, that PID:TOD identification can help to match the same mod_jk.log you're seeing, not to cross reference the mod_jk.log with the JBoss one. If you want to crossreference the two logs then you should find some other way, setting a custom header in Apache / mod_jk and printing that in your JBoss log is the first way i thought right now. Or at least you can check the requests around the failure time.

Fredi
  • 2,257
  • 10
  • 13
  • I thought the identifier was, [3732:2460]. I need to find that in the myJbossServer logs. Do I misunderstand here? – johnny Jan 26 '17 at 16:43
  • In short yes, i updated my answer with more info for that. Anyway, the info you need is the one in brackets, (myJbossServer) – Fredi Jan 26 '17 at 16:53
  • Should I be able to find the threadid in the logs of JBoss? – johnny Jan 26 '17 at 17:08
  • I don't know of a way to do that. But if you want to identify the failed request, just print the source port in both your frontend and your backend, that way you can match the two logs. This only if your backend is yet functioning, but as mod_jk went in error it may not even log it no? – Fredi Jan 26 '17 at 17:25
  • I did that. That's why I asked about the identifier, the threadid. I wanted something more specific. I can match up times, approximately, but I guess I misinterpreted RedHat's guide. – johnny Jan 26 '17 at 17:30
  • Indeed, as i readed it i clarified a bit my answer on this point. Actually i even decoded the traffic dump from a Apache -> JBoss system i have, no sign of that info so JBoss does not get it (even if possbile) by default. – Fredi Jan 26 '17 at 17:34