1

I have a Jenkins Maven job that has a post build script written in BASH. Periodically, while the post script is running, it seems like some sort of Maven operation is interfering with the BASH script causing the build to fail.

Here's some sample output from the Jenkins console

+ version_override='Downloading: http://clojars.org/repo/org/apache/maven/plugins/maven-metadata.xml
Downloading: http://clojars.org/repo/org/codehaus/mojo/maven-metadata.xml
Downloading: http://oss.sonatype.org/content/repositories/releases/org/apache/maven/plugins/maven-metadata.xml
Downloading: http://oss.sonatype.org/content/repositories/releases/org/codehaus/mojo/maven-metadata.xml

See how my BASH variable appears to be getting assigned a value from the Maven download? Now I won't say for certain that's what's going on. Maybe there's just two output streams going to the console simultaneously, one from Maven updating the repo, another from my BASH post build script. Either way, the Maven download seems to be interfering with my BASH script as it eventually crashes the build.

+ read config_path
+ '[' '!' -z http://clojars.org/repo/org/apache/maven/plugins/maven-metadata.xml ']'
+ http://clojars.org/repo/org/apache/maven/plugins/maven-metadata.xml
/usr/share/build-utils/lib/java.sh: line 115: http://clojars.org/repo/org/apache/maven/plugins/maven-metadata.xml: No such file or directory

That definitely looks like Maven output is creeping into my BASH script!

I'm almost positive this is Maven looking for updates. The strange thing is why would that run in parallel with a post build script. And even then, how is it possible that it would interfere with the runtime of said script? Truly bizarre!

Another reason I'm sure there's nothing wrong with my post build script. Every time this error comes up, I run the build again and it works. It's only when Maven is updating that the build fails.

I don't care if Maven wants to look for updates, but is there some way to configure it so that repo updates happen before my post build script?

quickshiftin
  • 2,125
  • 5
  • 27
  • 41
  • I could try to help you if you paste your config.xml for that jenkins job. – Danila Ladner Oct 05 '13 at 03:21
  • Ok, but it will have to wait until Monday, I don't have access to those systems outside the office. I'll post it up first thing on Monday morning. – quickshiftin Oct 05 '13 at 03:26
  • Thanks for your patience. Since the xml was close to 200 lines, I posted it on pastebin. http://pastebin.com/WfAuMAwJ – quickshiftin Oct 07 '13 at 16:01
  • Very interesting.., this morning builds started breaking from low perm size memory. I've changed _/etc/sysconfig/jenkins_, setting `JENKINS_JAVA_OPTIONS="-Djava.awt.headless=true -Xmx2048m -XX:MaxPermSize=512m"`; haven't had this problem come up in the last 5 builds. Fingers crossed this solved it, but the solution seems as bizarre as the problem! – quickshiftin Oct 07 '13 at 21:59
  • Yeah i was looking at your xml, is it possible for you also give me that script you run and full log output of running jenkins job? – Danila Ladner Oct 07 '13 at 22:20
  • The script is rather lengthy.., roughly 600 lines across several files. I can show you the [build output](http://pastebin.com/czPPPpLE) and [function where it's blowing up](http://pastebin.com/V68MhsCZ) from my script(s). Let me know if you need to see any more of the script code. – quickshiftin Oct 07 '13 at 22:32
  • Actually, [these](http://pastebin.com/PYHr9SDj) may be helpful, they're being called by the `deploy_properties` function from my last comment. One thing of note, when this does happen, the build bombs at the same location every time, so maybe there is something reproducible to track down; only I'm having a hard time to repro on purpose. – quickshiftin Oct 07 '13 at 22:35
  • Ok, so it breaks on evaluating the uri to maven-metadata.xml in your java.sh script. Can you please give me piece where that happens? I mean it is a weird, it might be that jenkins still updates the repo and is overwriting that file, while post-build trying to fetch that file. – Danila Ladner Oct 07 '13 at 23:12
  • The last part where it blows up `+ '[' '!' -z http://clojars.org/repo/org/codehaus/mojo/maven-metadata.xml ']'` is from [here](http://pastebin.com/061AAm1s). This bit `if [ ! -z "$4" ]; then`. So that part is after the call to [the first bit I posted](http://pastebin.com/V68MhsCZ) (`deploy properties`). It's making it through that part, then when it evaluates the 4th parameter it bombs. Thing is Jenkins seems to be stuffing the URL from Maven in there, here's my call to `package_java_project` - `package_java_project $submodule $java_project $version`; you can see, no 4th parameter passed.. – quickshiftin Oct 07 '13 at 23:21
  • Maybe if I just pass an explicit empty string `''` for the 4th parameter that would ensure `package_java_project` doesn't ever get any garbage from Jenkins? Seems like something I shouldn't have to do, but could be a successful workaround. – quickshiftin Oct 07 '13 at 23:22

1 Answers1

2

Finally figured it out! I'm using a Maven plugin to extract the module version via a Maven plugin, based on this SO article.

The command produces output like this

[INFO] --- maven-help-plugin:2.1.1:evaluate (default-cli) @ master-parent ---
[INFO] No artifact parameter specified, using 'com.myorg:master-parent:pom:0.3.02' as project.
[INFO] 
0.3.02
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]

My initial BASH call to extract the version number

version_override=$(mvn org.apache.maven.plugins:maven-help-plugin:2.1.1:evaluate -Dexpression=project.version | grep -v '[INFO]')

However, every so often (and yet unsure where to control this), Maven will reach out for updates to modules (I think). This is where the spurious Downloading output comes from, which mucks with my assignment from above.

I've now revised the command as follows

version_override=$(mvn org.apache.maven.plugins:maven-help-plugin:2.1.1:evaluate -Dexpression=project.version | egrep -v '[INFO]|Downloading:' | tr -d ' \n')

Likely still not bullet proof though, who knows what other output could come from this command, for example some day instead of leading [INFO] lines, maybe it spits out [ERROR] or something...

Anyway, it's tightened down for the time being. I really did think this was a Jenkins/Maven issue at first, but turns out it was just a BASH scripting issue!

Community
  • 1
quickshiftin
  • 2,125
  • 5
  • 27
  • 41
  • 1
    Looking through other ways to parse that out on the SO post, I've settled on `egrep -v '^\[|Downloading:' | tr -d ' \n'` for the filter which should be more flexible. – quickshiftin Oct 08 '13 at 20:40