1

I recently learned about withCredentials DSL thanks to an answer to this question. Having attempted to use @RamKamath's answer, i.e. the following Jenkinsfile:

pipeline {
agent any
stages {
stage( "1" ) {
  steps {
    script {
     def credId = "cred_id_stored_in_jenkins"
     withCredentials([usernamePassword(credentialsId: credId,
                                       passwordVariable: 'password',
                                       usernameVariable: 'username')]) {
      String url = "https://bitbucket.company.com/rest/build-status/1.0/commits"
      String commit = '0000000000000000000000000000000000000001'
      Map dict = [:]
      dict.state = "INPROGRESS"
      dict.key = "foo_002"
      dict.url = "http://server:8080/blue/organizations/jenkins/job/detail/job/002/pipeline"
      List command = []
      command.add("curl -f -L")
      command.add('-u ${username}:${password}')
      command.add("-H \\\"Content-Type: application/json\\\"")
      command.add("-X POST ${url}/${commit}")
      command.add("-d \\\''${JsonOutput.toJson(dict)}'\\\'")
                         
      sh(script: command.join(' '))
     }
    }
   }
  }
 }
}

...the curl command itself fails because of a reported "Bad request" error. This is the snippet from the Jenkins console output:

+ curl -f -L -u ****:**** -H "Content-Type:application/json" -X POST https://bitbucket.company.com/rest/build-status/1.0/commits/0000000000000000000000000000000000000001 -d '{"state":"INPROGRESS","key":"foo_002","url":"http://server:8080/blue/organizations/jenkins/job/detail/job/002/pipeline"}'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   153    0     0  100   153      0   4983 --:--:-- --:--:-- --:--:--  5100
curl: (22) The requested URL returned error: 400 Bad request

I understand that -u ****:**** is the masked username:password argument to -u.
If I copy/paste that exact string into a shell, and replace the masked values with the real values, the curl command works:

$ curl -f -L -u super_user:super_password -H "Content-Type:application/json" -X POST https://bitbucket.company.com/rest/build-status/1.0/commits/0000000000000000000000000000000000000001 -d '{"state":"INPROGRESS","key":"foo_002","url":"http://server:8080/blue/organizations/jenkins/job/detail/job/002/pipeline"}'
$ 

What is going wrong? Why does the curl command result in error 400/"Bad request" when Jenkins executes it, but the same command runs fine when executed manually?

Please note: as was recommended, I enclosed the -u ${username}:${password} in single-quotes, not double-quotes.


Update: I feel as though something is wrong with the string interpolation, because if I modify the Jenkinsfile to add a hardcoded username/password, i.e.

command.add('-u super_user:super_password')

...instead of

command.add('-u ${username}:${password}')

...then the curl command still fails exactly as before, i.e. because of the error: 400 Bad request

Can someone please help me identify what is wrong, presumably with the command assembly, and/or the sh() invocation?


Update

I've simplified the problem by removing the withCredentials(). Even this simplified curl invocation fails:

pipeline {
agent any
stages {
stage( "1" ) {
  steps {
    script {
     def credId = "cred_id_stored_in_jenkins"
     String url = "https://bitbucket.company.com/rest/build-status/1.0/commits"
     String commit = '0000000000000000000000000000000000000001'
     Map dict = [:]
     dict.state = "INPROGRESS"
     dict.key = "foo_002"
     dict.url = "http://server:8080/blue/organizations/jenkins/job/detail/job/002/pipeline"
     List command = []
     command.add("curl -f -L")
     command.add('-u super_user:super_password')
     command.add("-H \\\"Content-Type: application/json\\\"")
     command.add("-X POST ${url}/${commit}")
     command.add("-d \\\''${JsonOutput.toJson(dict)}'\\\'")

     sh(script: command.join(' '))
    }
   }
  }
 }
}
StoneThrow
  • 5,314
  • 4
  • 44
  • 86
  • `command.add(...)` is in the Groovy context. According to [_Groovy, Strings and GString literals, Double quoted string (interpolation)_](https://sodocumentation.net/groovy/topic/3409/strings-and-gstring-literals) and [_Groovy, String Interpolation_](https://sodocumentation.net/groovy/topic/3125/string-interpolation) what if you try `"..."` instead of `'...'` as string delimiters? If I try an adaption of your script with the single quotes inline a Pipeline project I get: `[curl -f -L, -u ${username}:${password}, -H \"Content-Type: application/json\", -X POST http://localhost:8083/api/json/]`. – Gerold Broser Aug 12 '21 at 22:58
  • @GeroldBroser - just gave it a try, i.e. now every instance of `command.add(...)` is `command.add("...")`. I.e. no instances of `command.add(...)` use single-quotes. But the `curl` command fails identically as in the original post. – StoneThrow Aug 12 '21 at 23:01
  • 1
    @GeroldBroser - Trying to simplify this problem to the simplest possible MVRE, I've discovered that the problem is reproducible even without the `withCredentials()` and variables -- i.e. everything with just hardcoded string! I will update my post accordingly; it might be helpful to inspect the simplest possible version of the problem. – StoneThrow Aug 12 '21 at 23:09
  • "_without the `withCredentials()`_" was one part of my "_adaption of your script_" already. :) – Gerold Broser Aug 12 '21 at 23:18
  • @GeroldBroser - updated with the `withCredentials()` removed. I'm pretty damned sure this some obscure little string interpolation issue starting me in the face. But for the life of me, I can't see it. The string-wrangling trying to get double-quotes in the right places in the command was hell, and I thought I'd nailed it -- again, because if I literally copy/paste the command from the Jenkins console output to a shell, the `curl` command works! – StoneThrow Aug 12 '21 at 23:31

2 Answers2

2

I tried it with an adaption of your script inline a Pipeline project:

pipeline {
    agent any
    stages {
        stage( "1" ) {
            steps {
                script {
                  def username = '********' // my real jenkins user
                  def password = '********' // my real jenkins pwd
                  String url = "http://localhost:8083/api/json"
                  String commit = ""
                  List command = []
                  command.add("'C:/Program Files/Git/mingw64/bin/curl.exe' -f -L")
                  command.add("-u ${username}:${password}")
                  command.add('-H "Content-Type: application/json"') // no "..." string delimiters and escaping necessary
                  //command.add("-X POST ${url}/${commit}")
                  command.add("-X GET ${url}/${commit}")  // GET instead
                  //command.add("-d \\\''${JsonOutput.toJson(dict)}'\\\'")
            
                  sh(script: command.join(' '))
                }
            }
        }
    }
}

Console Output

+ 'C:/Program Files/Git/mingw64/bin/curl.exe' -f -L -u ********:******** -H 'Content-Type: application/json' -X GET http://localhost:8083/api/json/
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  3232  100  3232    0     0  42667      0 --:--:-- --:--:-- --:--:-- 43675{"_class":"hudson.model.Hudson","assignedLabels":[{"name":"master"}],"mode":"NORMAL","nodeDescription":"the master Jenkins node","nodeName":"","numExecutors":2,"description":null,"jobs":
...

And I remember Jenkinsfile idiosynchrasies with escaping and quotes.

With Credentials Binding Plugin

        stage( "curl withCredentials" ) {
            steps {
                script {
                    withCredentials([usernamePassword(
                            credentialsId: 'jenkins-user',
                            passwordVariable: 'password',
                            usernameVariable: 'username')]) {
                                
                      String url = "http://localhost:8083/api/json"
                      String commit = ""
                      List command = []
                      command.add("'C:/Program Files/Git/mingw64/bin/curl.exe' -f -L")
                      command.add("-u ${username}:${password}")
                      command.add('-H "Content-Type: application/json"') // no "..." string delimiter and escaping necessary
                      //command.add("-X POST ${url}/${commit}")
                      command.add("-X GET ${url}/${commit}")  // GET instead
                      //command.add("-d \\\''${JsonOutput.toJson(dict)}'\\\'")
    
                      sh(script: command.join(' '))
                    }
                }
            }
        }            

Console Output

+ 'C:/Program Files/Git/mingw64/bin/curl.exe' -f -L -u ****:**** -H 'Content-Type: application/json' -X GET http://localhost:8083/api/json/
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  3231  100  3231    0     0  42247      0 --:--:-- --:--:-- --:--:-- 42513
100  3231  100  3231    0     0  42216      0 --:--:-- --:--:-- --:--:-- 42513{"_class":"hudson.model.Hudson","assignedLabels":[{"name":"master"}],"mode":"NORMAL","nodeDescription":"the master Jenkins node","nodeName":"","numExecutors":2,"description":null,"jobs":
...
Gerold Broser
  • 14,080
  • 5
  • 48
  • 107
  • Still failing...I notice you're testing on Windows, and this makes me wonder if the issue is some subtle difference in how Windows interpolates strings vs. Linux Bash. Adding `-v` to the `curl` command looks promising: when Jenkins runs the curl command, it reports `Content-Length: 143`, but when I copy/paste that command from the Jenkins console output to a `bash` shell, then `curl` reports `Content-Length: 141`. So there is some difference between what Jenkins is "really" executing and what it reports in the console log. – StoneThrow Aug 12 '21 at 23:54
  • @StoneThrow The string interpolation is done inside Jenkins already. See what it prints as cmd line. There's nothing to interpolate any more. (And, BTW, MinGW's and Linux' Bash aren't that different when it comes to String interpolation. The only difference I ever experienced was different behaviour of the same program option/parameter.) – Gerold Broser Aug 13 '21 at 00:04
  • Holy hell: guess what `curl --trace-ascii /dev/stdout ...` revealed: when _Jenkins_ executes the `curl command` the HTTP POST content _includes a leading and trailing single-quote_ (before and after the json content)!!! This is weird - I don't know how to "describe" this problem: with all that crazy escape-sequencing, what _Jenkins_ ends up actually executing includes the leading and trailing single-quote _in the HTTP POST content`...yet I copy/paste that entire command as printed in the Jenkins console output into a Bash shell, and it works! – StoneThrow Aug 13 '21 at 00:14
  • ...so I think this problem degenerates into a Jenkins string-wrangling problem...which I may ask as a separate question, in order to not clutter this one. – StoneThrow Aug 13 '21 at 00:15
  • @StoneThrow Have you seen the link to the gist I added in between? – Gerold Broser Aug 13 '21 at 00:17
  • 1
    I'm checking the link out now. I was hoping the problem was specifically _not_ string-wrangling/escaping, because prior experience has taught me the harsh lesson of how harrowing that is in Jenkins! :P – StoneThrow Aug 13 '21 at 00:18
  • 1
    At last! A working solution! That was a nightmarish experience! :P Your help was invaluable - thank you. I don't mean to discount your answer by posting my own, but my working solution was different than yours, because of what I can only chalk up to a difference between the Windows and Unix/bash environments. Take a look at my answer if you're interested, and I hope it's properly comprehensive and provides some good insights. – StoneThrow Aug 13 '21 at 13:15
1

This issue turned out to be a string-escaping problem. The working solution -- including withCredentials(), which was not a factor in the problem -- for me was:

pipeline {
 agent any
  stages {
    stage( "1" ) {
      steps {
        script {
          def credId = "cred_id_stored_in_jenkins"
          String url = "https://bitbucket.company.com/rest/build-status/1.0/commits"
          String commit = '0000000000000000000000000000000000000001'
          withCredentials([usernamePassword(credentialsId: credId,
                                            passwordVariable: 'password',
                                            usernameVariable: 'username')]) {
            Map dict = [:]
            dict.state = "INPROGRESS"
            dict.key = "foo_002"
            dict.url = http://server:8080/blue/organizations/jenkins/job/detail/job/002/pipeline"
            def cmd = "curl -f -L" +
                      "-u ${username}:${password} " +
                      "-H \"Content-Type: application/json\" " +
                      "-X POST ${url}/${commit} " 
                      "-d \'${JsonOutput.toJson(dict)}\'")
                         
            sh(script: cmd)
          }
        }
      }
    }
  }
}

I'm sure some variation of the List.join() would have worked - and there's no specific reason I reverted to using + to join the strings other than I was hacking away, and settled on the first thing that just worked. Escaping strings in Jenkins appears to be its own little circle of Hell, so I don't want to spend more time there than I need to.

A few oddities revealed themselves while working on this:

First, behavior appears to be different in Windows vs. Unix/bash: @GeroldBroser (whose help was invaluable) was able to get a working solution in his Windows environment with string-escaping closer/identical to my original post; however I was not able to reproduce his result in my Unix/bash environment (Jenkins sh invocations use bash in my setup).

Lastly, I was under the impression that the text logged to a Jenkins job console output was literally what was executed -- but this doesn't appear to be quite true.
To summarize a portion of my comment-discussion with @GeroldBroser:
The curl command, when run by Jenkins failed with error: 400 Bad request, yet if I copy/pasted/executed the exact curl command logged in my Jenkins job console ouput in a bash shell, it worked successfully.
By making use of the --trace-ascii /dev/stdout option to curl, I was able to discover that the curl command, when run successfully in bash, sent 141 bytes, but when run unsuccessfully by Jenkins, sent 143 bytes: the extra 2 bytes were leading and trailing ' (single-quote) characters before and after the JSON content.
This led me down the path of madness to the circle of Hell to the castle of damnation to the throne of insanity that is Jenkins string escaping, and I eventually arrived at the above working solution.

Noteworthy: with this working solution, I can no longer copy/paste the curl command -- as logged in my Jenkins job console output -- to a bash shell and successfully execute. Therefore, it's not (always) true that "what is logged in the Jenkins job console output is exactly what is run (i.e. copy/pastable) in the shell."

StoneThrow
  • 5,314
  • 4
  • 44
  • 86