10

I am trying to configure Jenkins to work with Atlassian Stash. I have gotten as far as (apparently) getting the Git plugin to successfully ping Stash, as there are no longer any error messages from the Git plugin page. I am using Git with SSH, and I have configured my Stash account with the correct SSH key. I should also mention that my office is behind a firewall, hence the proxy server for Stash. Here is a screenshot of the Jenkins Git configuration:

enter image description here

The console output indicates failure after the default 10 minute timeout. Here is what the console output says:

 > C:\Program Files (x86)\Git\bin\git.exe fetch --tags --progress ssh://git@stash.someproxy.com:7999/project/myrepo.git 
+refs/heads/*:refs/remotes/origin/*
ERROR: Timeout after 10 minutes
ERROR: Error cloning remote repo 'origin'
hudson.plugins.git.GitException: Command "C:\Program Files (x86)\Git\bin\git.exe fetch --tags --progress ssh://git@stash.someproxy.com:7999/project/myrepo.git +refs/heads/*:refs/remotes/origin/*" returned status code -1:
stdout: 
stderr: 
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn
    ...

When I run the following Git command directly from the Windows bash, it completes quickly, although there is no output:

$ time git fetch --tags --progress 
ssh://git@stash.someproxy.com:7999/pcb_imst/
wigbotsimulator.git +refs/heads/*:refs/remotes/origin/*

real    0m5.530s
user    0m0.093s
sys     0m0.062s

Also, when I remove the credentials from Jenkins and try to build, I get an authentication error almost immediately (Permission denied (publickey)). So it definitely seems that Jenkins can hit Stash, though it does not appear to be able to complete a fetch.

I am out of ideas on how to fix this. It is strange that Git works locally but does not work when Jenkins uses it.

Tim Biegeleisen
  • 502,043
  • 27
  • 286
  • 360
  • Does a ssh -T git@stash.someproxy.com -p 7999 authenticate you (meaning you get some kinf=d of welcome message stating your account?) – VonC Feb 05 '18 at 05:37
  • @VonC It just hangs from what I can see. – Tim Biegeleisen Feb 05 '18 at 05:46
  • Can you try to use the IP for that server, without any reference ot a proxy? – VonC Feb 05 '18 at 05:51
  • @VonC I can't get anywhere with `ssh -T git@xxx`. When I tried doing `ssh -T git@stash.someproxy.com -p 7999` I got the response `no address associated with name`. But can you explain why, from the same console, I can talk to Stash via SSH? I just verified that `git fetch` worked using the SSH certificates. – Tim Biegeleisen Feb 05 '18 at 07:01
  • Does it work (in command line) for git clone as well. By the way, do you need credentials at all? If your public key is registered, and your private key does not have any passphrase, you should need no additional credentials. – VonC Feb 05 '18 at 07:05
  • @VonC OK so shall I clone into some new random directory, just to see if that works? Yes, the cert has no passphrase. – Tim Biegeleisen Feb 05 '18 at 07:06
  • Yes, just for testing, clone into another (empty) path. – VonC Feb 05 '18 at 07:07
  • Cloning into a new empty folder, outside of the other Git folders, completely worked. Command used: `git clone ssh://git@stash.someproxy.com:7999/.../project.git` – Tim Biegeleisen Feb 05 '18 at 07:09
  • I suppose you have indicated the right ssh key to use somehow? (~/.ssh/config? GIT_SSH_COMMAND?) Unless it is a public repo?. – VonC Feb 05 '18 at 07:10
  • I have a small script in both my `.bashrc` and `.project` files in my Windows user folder. I got these scripts from the GitHub website, for the purpose of enabling the SSH agent at startup, see here: https://gist.github.com/bsara/5c4d90db3016814a3d2fe38d314f9c23 – Tim Biegeleisen Feb 05 '18 at 07:13
  • I'm certain that SSH is working from the Git bash, but I'm at a total loss about why it fails for Jenkins. – Tim Biegeleisen Feb 05 '18 at 07:14
  • No agent should be required if you don't have an associated passphrase. Is the name of your ssh key the default one (%USERPROFILE%\.ssh\id_rsa?) – VonC Feb 05 '18 at 07:15
  • Yes; I have four files in the `.ssh` folder directly under my user profile: `agent.env`, `id_rsa`, `id_rsa.pub`, and `known_hosts`. Oh, and I should tell you that I pasted the contents of `id_rsa.pub` into Stash directly as a key. – Tim Biegeleisen Feb 05 '18 at 07:15
  • But does your private key has an associated passphrase though? If not, remove any reference to credentials in your Stash settings – VonC Feb 05 '18 at 07:22
  • @VonC I just ran `ssh-keygen -y` from the folder containing both the public and private key, and I was _not_ prompted for a passphrase. Also of note, I ran `ssh-keygen -y -e -f id_rsa` and got the cert printed to the console as output. – Tim Biegeleisen Feb 05 '18 at 07:26
  • OK, so no passphrase. Did you try and remove the credentials from Stash settings? Also, no need for any agent of any kind. – VonC Feb 05 '18 at 07:27
  • Yes, I have gone over this at least of handful of times. The key currently in Stash is the 3rd or 4th attempt. I do not encounter issues when removing the keys from Stash (though I would not expect having _extra_ keys in Stash would cause authentication problems for a given valid key). In any case, what is not explained well here is why SSH works from the Bash but fails from Jenkins. A good thing to try would be to test a third approach to connecting with Stash. – Tim Biegeleisen Feb 05 '18 at 07:28
  • Next, make sure to use the system git, not the embedded one. And make sure your %PATH% reference that system Git, which should be the latest one available https://github.com/git-for-windows/git/releases – VonC Feb 05 '18 at 07:30
  • @VonC Apologies for the delay, I had to restart my VM after adding Git to the PATH. I have done this; running `git --version` from anywhere is now recognized. As for the Git being used by Jenkins, it is the same one which I am using from the Bash: `C:\Program Files (x86)\Git\bin\git.exe`. Is there some sanity check we can do from within Jenkins to make sure it can access Git? – Tim Biegeleisen Feb 05 '18 at 07:53
  • The main check is: what Jenkins account is used to run Jenkins? The system account (as a service) or your own? – VonC Feb 05 '18 at 07:56
  • I nkonw understand the credentials (I though they were for Stash, but they are actually Jenkins') Make sure those credentials are SSH ones, referencing your private key. – VonC Feb 05 '18 at 07:57
  • @VonC Brilliant. You may have just hit on the problem. _Whose_ account is my Jenkins using, and therefore _whose_ SSH stuff is it using? And, most important, how do I find this out? – Tim Biegeleisen Feb 05 '18 at 07:59
  • Is Jenkins running as a Windows service? If yes, open that services (type `services.msc`, look for Jenkins), and check the user tab. – VonC Feb 05 '18 at 08:06
  • I don't have a user tab from my Windows Services window. What I do see is that "Log On" is happening via the Local System account, rather than "This account." – Tim Biegeleisen Feb 05 '18 at 08:08
  • Using your own account would be best. Although the credentials should be able to indicate the proper key to use. Provided you are using the Jenkins SSH credentials plugin (https://wiki.jenkins.io/display/JENKINS/SSH+Credentials+Plugin), not just the credentials one. – VonC Feb 05 '18 at 08:10
  • I will but I cannot put an answer before several hours: I am at work right now. – VonC Feb 05 '18 at 08:12
  • Hi Von, I already have the SSH credentials plugin installed. Can you paste a link which discusses how to change/find out what is the Jenkins Windows user? – Tim Biegeleisen Feb 05 '18 at 08:18
  • https://jenkins-le-guide-complet.github.io/html/sect-windows-service.html: see the picture with "This account" – VonC Feb 05 '18 at 08:21
  • "I already have the SSH credentials plugin installed." But in the picture, I see Credentials "biegelet": Is "biegelet" an entry in your SSH Jenkins section, where you reference explicitly your SSH private key. Or is is a classic credentials with username/password? – VonC Feb 05 '18 at 08:23
  • No... of course `biegelet` refers to an SSH key based credential, and _not_ a username and password. I just happened to name it with my username, that's all. – Tim Biegeleisen Feb 05 '18 at 08:26
  • Then the service account should not be important, considering that credential should mention the full path of the private key to use. – VonC Feb 05 '18 at 08:27
  • @VonC Yes, it uses a private key with the path `F:\.ssh\id_rsa` (don't ask...I work for a bank, and everyone uses a VM...this is my home directory). Anyway, the Windows user under which Jenkins is running is `NT AUTHORITY\SYSTEM`. I poked around to learn that this is a system account, very powerful, and not very configurable. It seems to good to have Jenkins runs as this, because then it would have unfettered ability to do anything it needs to do. – Tim Biegeleisen Feb 05 '18 at 08:29
  • I won't ask ;) same here. However, the Jenkins logs does not show that key as being used though... – VonC Feb 05 '18 at 08:31
  • @VonC I get a periodic error from the project configuration page under the "Source Code Management" tab, complaining that it can't reach the Git server. The error is in red, and indicates that Jenkins can't seem to connect via SSH. – Tim Biegeleisen Feb 05 '18 at 08:34
  • @VonC My boss just dropped by my desk, and forced me to try HTTPS again. To my shock, it worked. Now I'm really confused. Everything works everywhere, except Jenkins with SSH. – Tim Biegeleisen Feb 05 '18 at 08:53
  • @Did you enter your username/password at any point when using an HTTPS URL? – VonC Feb 05 '18 at 09:00
  • No...I created a username/password credential using the plugin, and I used that. – Tim Biegeleisen Feb 05 '18 at 09:01
  • That will work too ;) (using the classic Jenkins Credentials plugin, this time). I will post an answer later today. – VonC Feb 05 '18 at 09:01

2 Answers2

4

After conversing extensively with @VonC and no glaring errors having turned up, I made an attempt to have Jenkins connect via HTTPS. Initially I had abandoned HTTPS because it too was not working. To my surprise, the HTTPS build worked without error.

I was getting ready to give up on SSH, when, as a last resort, I decided to use the following private key option:

enter image description here

In other words, I cut and pasted my private key (~/id_rsa) directly into Jenkins. To my amazement, this worked! I am working off a VM, and my user folder is on an F: drive, which apparently Jenkins was not able to read.

So the root cause of my problem was in fact a problem with the SSH private key file. But the exact error was that this private key was not readable by Jenkins. A lesson learned is to make sure that Jenkins has the ability to read every file which it needs to function. Just because your Git setup is working locally does not mean it will also be working for Jenkins.

Tim Biegeleisen
  • 502,043
  • 27
  • 286
  • 360
4

As discussed, using the Jenkins SSH credential plugins is the right solution.
Copying directly the private SSH key is easier than relying on Jenkins master to know where to find ~/.ssh/id_rsa (on Windows, %USERPROFILE%\.ssh\id_rda), especially considering Jenkins often used with the System account, not the local one.

Stash has the public key. If Jenkins has the private key, it will be able to complete the authentication.

HTTPS would work too, considering the credential manager is the stash one, which has already cached the username/password.

VonC
  • 1,262,500
  • 529
  • 4,410
  • 5,250
  • I'm awarding you the bounty because I could argue that the train of thought we had led me to stumble onto the right answer. And your suggestion to use HTTPS as an alternative to SSH is a valid one, and also worked for me. – Tim Biegeleisen Feb 13 '18 at 04:33
  • @TimBiegeleisen Thank you. I missed your own answer at the time. +1 – VonC Feb 13 '18 at 06:01