1

In our jenkins build configuration we have the following shell commands (Execute shell) as the last step of our build process. 

echo "$USER"
echo "Script executed from: ${PWD}"
git submodule update --init --recursive
if [ -f composer.json ]; then
    /usr/local/bin/composer install
fi
sed -i 's/phpunit --bootstrap/\/usr\/local\/bin\/phpunit \-\-bootstrap/g' runtest
./runtest --config unittest/configure.conf --coverage /var/lib/jenkins/workspace/coverage/

Everything used to work perfectly till our hard drive gets full and jenkins stopped building. After we cleaned the hard drive and tried building through jenkins again, the last step (Execute shell) failed to build with the following error:

[project] $ /bin/sh -xe /tmp/hudson815776538860444797.sh
+ echo jenkins
jenkins
+ echo 'Script executed from: /var/lib/jenkins/workspace/project'
Script executed from: /var/lib/jenkins/workspace/project
+ git submodule update --init --recursive
error: git-submodule died of signal 9
Build step 'Execute shell' marked build as failure

We tried running the shell commands as root and everything works as expected. However, once we run it as jenkins user, the process gets killed within few seconds. 

[root@localhost project]# su -c "/bin/bash -x /tmp/hudson5288327457846724302.sh" -s /bin/sh jenkins
+ echo jenkins
jenkins
+ echo 'Script executed from: /var/lib/jenkins/workspace/project'
Script executed from: /var/lib/jenkins/workspace/project
+ '[' -f composer.json ']'
+ /usr/local/bin/composer install
Killed

I checked the system logs /var/log/messages but didnt find any clue. What can cause the issue and how we may troubleshoot this?

user3360140
  • 281
  • 1
  • 4
  • 14
  • This can be some kind of resource limitation (ulimit?). You can check `dmesg` output to see if there's any suspicious kernel message. You could also try more verbose git logging via, for example, `GIT_TRACE=2` (see https://stackoverflow.com/questions/6178401/how-can-i-debug-git-git-shell-related-problems for more details). – Juraj Martinka Apr 22 '19 at 07:37
  • Hey @JurajMartinka thanks a lot for your response. I did check `dmesg` but didnt find anything interesting. WIll check my `ulimit`. But, I how does getting git log help in this case? We are not doing anything with git in the script above? – user3360140 Apr 23 '19 at 17:17
  • I see `git submodule update --init --recursive` in the script and `git-submodule` is the command that died so I assumed (although it's a long shot) that verbose logging could provide more details about what's going on. – Juraj Martinka Apr 23 '19 at 19:35
  • I see, sorry I should have clarified. We experience the same issue even if we remove the `git submodule update` line from the script – user3360140 Apr 23 '19 at 20:35

1 Answers1

0

What is the state of your machine, specifically memory:

free -m

It could be the OOM killer. You would see logs. Try

grep -i “killed process” /var/log/messages
edzzz
  • 11
  • 1