So I've recently ran into weird problem, my jenkins build fails every odd build for no reason. Build is simple, it runs ansible playbook which in turns deploys code to remote machines, clears caches, runs maintenance, etc.
Build basically runs this:
ansible-playbook -i ansible/hosts ansible/backend.yml
Then, when the build number is odd, it fails on this step(runs with force=yes so local modifications should be no problem):
TASK [backend : Clone backend repo] ********************************************
task path: /opt/provision/ansible/roles/backend/tasks/main.yml:5
Using module file /usr/lib/python2.6/site-packages/ansible/modules/core/source_control/git.py
<server_ip_address> ESTABLISH SSH CONNECTION FOR USER: username
<server_ip_address> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o 'IdentityFile="/var/lib/jenkins/.ssh/jenkins"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=username -o ConnectTimeout=10 -o ConnectionAttempts=20 -o ControlPath=/var/lib/jenkins/.ansible/cp/ansible-ssh-%h-%p-%r server_ip_address '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1485853855.83-181127458563499 `" && echo ansible-tmp-1485853855.83-181127458563499="` echo $HOME/.ansible/tmp/ansible-tmp-1485853855.83-181127458563499 `" ) && sleep 0'"'"''
<server_ip_address> PUT /tmp/tmpdgsmCK TO /home/username/.ansible/tmp/ansible-tmp-1485853855.83-181127458563499/git.py
<server_ip_address> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o 'IdentityFile="/var/lib/jenkins/.ssh/jenkins"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=username -o ConnectTimeout=10 -o ConnectionAttempts=20 -o ControlPath=/var/lib/jenkins/.ansible/cp/ansible-ssh-%h-%p-%r '[server_ip_address]'
<server_ip_address> ESTABLISH SSH CONNECTION FOR USER: username
<server_ip_address> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o 'IdentityFile="/var/lib/jenkins/.ssh/jenkins"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=username -o ConnectTimeout=10 -o ConnectionAttempts=20 -o ControlPath=/var/lib/jenkins/.ansible/cp/ansible-ssh-%h-%p-%r server_ip_address '/bin/sh -c '"'"'chmod u+x /home/username/.ansible/tmp/ansible-tmp-1485853855.83-181127458563499/ /home/username/.ansible/tmp/ansible-tmp-1485853855.83-181127458563499/git.py && sleep 0'"'"''
<server_ip_address> ESTABLISH SSH CONNECTION FOR USER: username
<server_ip_address> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o 'IdentityFile="/var/lib/jenkins/.ssh/jenkins"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=username -o ConnectTimeout=10 -o ConnectionAttempts=20 -o ControlPath=/var/lib/jenkins/.ansible/cp/ansible-ssh-%h-%p-%r -tt server_ip_address '/bin/sh -c '"'"'/usr/bin/python /home/username/.ansible/tmp/ansible-tmp-1485853855.83-181127458563499/git.py; rm -rf "/home/username/.ansible/tmp/ansible-tmp-1485853855.83-181127458563499/" > /dev/null 2>&1 && sleep 0'"'"''
fatal: [backend]: FAILED! => {
"changed": false,
"failed": true,
"invocation": {
"module_name": "git"
},
"module_stderr": "OpenSSH_6.6.1, OpenSSL 1.0.1k-fips 8 Jan 2015\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 56: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 23456\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 4\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\nShared connection to server_ip_address closed.\r\n",
"module_stdout": "Traceback (most recent call last):\r\n File \"/tmp/ansible_lVW4UB/ansible_module_git.py\", line 1040, in <module>\r\n main()\r\n File \"/tmp/ansible_lVW4UB/ansible_module_git.py\", line 994, in main\r\n result.update(changed=True, after=remote_head, msg='Local modifications exist')\r\nUnboundLocalError: local variable 'remote_head' referenced before assignment\r\n",
"msg": "MODULE FAILURE"
}
to retry, use: --limit @/opt/provision/ansible/backend.retry
PLAY RECAP *********************************************************************
backend : ok=2 changed=1 unreachable=0 failed=1
Traceback formated:
Traceback (most recent call last):
File \"/tmp/ansible_lVW4UB/ansible_module_git.py\", line 1040, in <module>
main()
File \"/tmp/ansible_lVW4UB/ansible_module_git.py\", line 994, in main
result.update(changed=True, after=remote_head, msg='Local modifications exist')
UnboundLocalError: local variable 'remote_head' referenced before assignment
Failing step:
- name: Clone backend
git: >
repo="{{ backend_repo }}"
dest="{{ backend_path }}"
accept_hostkey=yes
key_file="{{ backend_key }}"
clone=yes
force=yes
update=yes
version={{ backend_branch }}
If you run build immediately after it fails, it builds nicely. Also it builds every time I run it from command line.
