21

I have been scouring the internet for days, I have a problem similar to this.

I need to retrieve the console output in raw (plain) text. But if I can get it in HTML that is fine too, I can always parse it. The only thing is that I need to get it during the build step, which is a problem since the location where it should be available is truncated...

I have tried retrieving the console output from the following URL's (relative to the job):

  • /consoleText
  • /logText/progressiveText
  • /logText/progressiveHTML

The two text ones are plain text and would be perfect if not for the truncation, same goes for the HTML one... exactly what I need - only its truncated....

I am sure it is possible to retrieve this information somehow, since when viewing /consoleFull there is a real-time update of the console, without truncating or buffering.

However, upon examining that web page, instead of finding the content I desired, I found this code where it should have been (I did not include the full pages code, since it would be mostly irrelevant, and I believe those answering would be able to find out and know what should be there on their own)

      new Ajax.Request(href,{
          method: "post",
          parameters: {"start":e.fetchedBytes},
        requestHeaders: headers,
          onComplete: function(rsp,_) {

          var stickToBottom = scroller.isSticking();
          var text = rsp.responseText;
          if(text!="") {
            var p = document.createElement("DIV");
            e.appendChild(p); // Needs to be first for IE
            // Use "outerHTML" for IE; workaround for:
            // http://www.quirksmode.org/bugreports/archives/2004/11/innerhtml_and_t.html
            if (p.outerHTML) {
              p.outerHTML = '<pre>'+text+'</pre>';
              p = e.lastChild;
            }
            else p.innerHTML = text;
            Behaviour.applySubtree(p);
            if(stickToBottom) scroller.scrollToBottom();
          }

          e.fetchedBytes     = rsp.getResponseHeader("X-Text-Size");
          e.consoleAnnotator = rsp.getResponseHeader("X-ConsoleAnnotator");
            if(rsp.getResponseHeader("X-More-Data")=="true")
              setTimeout(function(){fetchNext(e,href);},1000);
          else
              $("spinner").style.display = "none";
          }
      });

Specifically, I am hoping there is a way for me to get the content from text whatever it may be. I am not familiar with this language and so am not sure how I might be able to get the content I want. Plugins won't help since I want to retrieve this content as part of my script during the build step

Community
  • 1
  • 1
Inbar Rose
  • 41,843
  • 24
  • 85
  • 131

3 Answers3

24

You did pretty much good investigation already. I can only add the following: all console related plug-ins I know are designed as a post build actions.

The Log Trigger plugin provides a post-build action that allows Hudson builds to search their console log for a given regular expression and if found, trigger additional downstream jobs.

So it looks like there is no straightforward solution to your problem. I can see the following options:

1. Use tee or something similar (applicable to shell build steps only)

This solution is far from being universal, but it can provide quick access to the latest console output, produced by a command or set of command.

tee - read from standard input and write to standard output and files

Using synonyms on the system level other Jenkins build steps can modified in order to produce console output. File with console output can be referenced through Jenkins or using any other way.

2. Modify Jenkins code

You can just do a quick fix for internal usage or provide a patch introducing specific system-wide setting.

3. Mimic /console behavior

Code in your example is used to request updates from the Jenkins server. As you may expect the server side can return piece of information starting with some offset. Let me show.

Periodically console page sends requests to the server:

enter image description here

Parameters are straightforward:

enter image description here

Response is a chunk of information to be added:

enter image description here

Another request with updated offset (start) value

enter image description here

You can easily understand there is no data by analyzing Content-Length

enter image description here

So the answer is: use url/job-name/build-number/logText/progressiveHtml, specify start offset, send request and receive console update.

Renat Gilmanov
  • 17,735
  • 5
  • 39
  • 56
  • I meant to use this url: url/job-name/build-number/logText, send POST request and receive new chunk of console output. – Renat Gilmanov Jul 29 '13 at 14:36
  • Sorry, I've updated the answer, the correct URL is url/job-name/build-number/logText/progressiveHtml (please check my screens, Chrome shows all details). – Renat Gilmanov Jul 29 '13 at 14:50
  • Yes, this is the only way, since you want to analyze output in real-time. So you can control polling period and do some other fancy stuff. Moreover, it simplifies HUGE log handling. Please also consider some significant delays are possible. – Renat Gilmanov Jul 29 '13 at 14:57
  • 1
    Excellent! This works fine, one note though - I do not get a `"Content-Length"` response, I get a `"X-Text-Size"` which is pretty much the same. This lets me read the data in chunks, thanks! :) - still need some more testing to see if this really works. – Inbar Rose Jul 29 '13 at 14:57
  • After testing, this solution worked wonderfully. Thank you for your time, you deserve the rep :) – Inbar Rose Jul 30 '13 at 10:03
  • This was extremely helpful! Though I wonder if one can also do the progressive log parsing without the Html? IE get the raw console output in a progressive manner. – S.Richmond Apr 23 '16 at 23:46
  • How do you know the end of the file/output though? Is it possible to get `content-length: 0` but a later request results in some content? – Obay Abd-Algader Apr 06 '22 at 20:24
3

I had a similar issue, the last part of my Jenkinsfile build script needs to parse the ConsoleLog for particular error messages to put in an email build report.

First attempt: http request.
It felt like a hack, it mostly worked, but ran into issues when we locked down access to the Jenkins server & my build nodes could no longer perform annon http gets on the page

Second attempt: use the APIs to enumerate the log lines.
It felt like the right thing to do, but it failed horribly as my nodes would take 30 minutes to get through the 100 meg log files. My presumption is that the Jenkins server was not caching the file, so each request involved a re-reading of the entire file up until the point of the last read.

Third and most successful solution: run grep on the server. 

node('master') {  
  sh 'grep some_criteria $JENKINS_HOME/workspace/path/to/job/console.log'  
}

it was fast, reliable, and it didn't matter how big the log files were.

Yes, this required trust of the Jenkins admin and knowledge of the directory paths on the Jenkins server - but since I was the admin, I trusted myself to do the right thing. Your mileage may vary.

Jason De Arte
  • 467
  • 1
  • 3
  • 14
2

To add some insight: when the Jenkins build was in progress, the response for the .../consoleText URL maxed out at 10000 lines, exactly. I was using 'requests()' package in Python. I have tried the same URL with curl and again received only the first 10K lines. Only after the build has finished both methods returned the full log (>22K lines in my case).

I will research further and hope to report back.

[2015-08-18] Update: It seems that this is a known issue (see here) and it's fixed in Jenkins 1.618 and later. I am still running 1.615 so I cannot verify. Amir

Amir Katz
  • 643
  • 2
  • 10
  • 23