Generate GitLab Merge Request when a push comes in via CI/CD Pipeline using Bash and cURL

Question

So, I've been beating my head around this for a few days and could use some help. Initially, I found the gitlab blog post from 2017: https://about.gitlab.com/blog/2017/09/05/how-to-automatically-create-a-new-mr-on-gitlab-with-gitlab-ci/

However it seems like a lot of it is outdated(?) but I was able to use POSTMAN to carve down a working model (using token as a parameter instead of a header, etc.)

Here's the thing though, I cannot get the Merge Request to trigger and I believe it's due to --data (-d) not interpolating the variables.

I've hardcoded the payload with the values exactly, and pushed to my branch and it generated the MR just fine.

But any method I use to try and incorporate variables will result in the cURL going through, but no MR being generated.

What's more, the GL Runner does not output the response - so it's difficult for me to figure out exactly what the issue is.

I've tried escaping with back slashes, I've tried single quote setups with escaping the single quotes and doing inner double quotes with the ${} variables, but nothing.

I've validated that the variables being sent in from the yml exists (echoing them out before the curl call and the body setup), but I'm just lost at this point.

I've still yet to try JQ, but I'm not sure if the runner has it installed, but I'll find out when I return back to test.

Any help would be lovely.

General format of the scripts code:

#!/usr/bin/env bash
TARGET_BRANCH="test"

echo "Host: ${HOST}"
echo "Project  Id: ${CI_PROJECT_ID}
echo "SOURCE BRANCH: ${CI_COMMIT_REF_NAME}
echo "Target Branch: ${TARGET_BRANCH}

BODY="{
    \"id\": ${CI_PROJECT_ID},
    \"source_branch\": \"${CI_COMMIT_REF_NAME}\",
    \"target_branch\": \"${TARGET_BRANCH}\",
    \"remove_source_branch\": false,
    \"title\": \"WIP: ${CI_COMMIT_REF_NAME}\",
}";

echo "Body: ${BODY}"

# Create MR
curl -X POST https://${HOST}/api/v4/projects/${CI_PROJECT_ID}/merge_requests?private_token${PRIVATE_TOKEN}"\
--header "PRIVATE_TOKEN: ${PRIVATE_TOKEN}"\
--header "Content-Type: application/json"\
--data "${BODY}"

Now I want to note, that after testing this and seeing the curl go through, but no MR generated, I modified the --data value to go from ${BODY} to: (assume to be valid value, just obscuring for reasons)

--data '{
  "id": <x>,
  "source_branch": "dev",
  "target_branch": "test",
  "remove_source_branch": false,
  "title":"WIP: dev"
}'

And this worked! I the MR generated just fine.

So I I then tested by replacing the single quote with double quotes, and escaping the inner double quotes but still keeping the hardcoded values. Back to no MR being generated.

I'm quote stumped - I even went back to the BODY=... block, and tried to reformat that.

Then I went to the outer single, inner double, escaping

'{
  "id": '"${CI_PROJECT_ID}"',
  ...
'}

and still nothing.

So I'm taking a break after all the attempts and speaking with some teammates to no avail, and throwing a hail marry Stack Overflow for some advice.

Added note: This is not a typical application project repo, so any discussion on "this isn't a good standard practice, use issues as an anchor point for generating MRs" doesn't apply when the use of the repo will not be using issues, have minimum user hands on, etc.

I value those kinds of insights, but again, the usecase for this repo is not an actual "project", so to speak.

Thank you

score 0 · Answer 1 · answered Sep 19 '22 at 17:56

I'm a bit confused about what it is you're asking. It sounds like you got it to work just fine by passing the json string directly to curl's data flag, and you're just wondering why the other attempts didn't work. If that's the case, here's what I see which would make your other attempts fail:

# ...
BODY="{
    \"id\": ${CI_PROJECT_ID},
    \"source_branch\": \"${CI_COMMIT_REF_NAME}\",
    \"target_branch\": \"${TARGET_BRANCH}\",
    \"remove_source_branch\": false,
    \"title\": \"WIP: ${CI_COMMIT_REF_NAME}\",
}";

echo "Body: ${BODY}"

# Create MR
curl -X POST https://${HOST}/api/v4/projects/${CI_PROJECT_ID}/merge_requests?private_token${PRIVATE_TOKEN}"\
--header "PRIVATE_TOKEN: ${PRIVATE_TOKEN}"\
--header "Content-Type: application/json"\
--data "${BODY}"

In this first option, the variable looks just fine, but when you're using it in your curl command, you're quoting the variable (ie, "$BODY"). There's nothing wrong with this when you're doing concatenation, but here it's likely causing a syntax issue since when the variable is extracted, the result is a json string that's doubly quoted:

""{
    \"id\": ${CI_PROJECT_ID},
    \"source_branch\": \"${CI_COMMIT_REF_NAME}\",
    \"target_branch\": \"${TARGET_BRANCH}\",
    \"remove_source_branch\": false,
    \"title\": \"WIP: ${CI_COMMIT_REF_NAME}\",
}""

I would try passing the $BODY variable without any quotes:

# Create MR
curl -X POST https://${HOST}/api/v4/projects/${CI_PROJECT_ID}/merge_requests?private_token${PRIVATE_TOKEN}"\
--header "PRIVATE_TOKEN: ${PRIVATE_TOKEN}"\
--header "Content-Type: application/json"\
--data $BODY

For the third attempt, I assume there's a typo in your question since you have what should be the final single-quote in front of the closing brace. The other issue is that you're terminating the single-quoted string in the middle of the json:

'{
  "id": '"${CI_PROJECT_ID}"',
  ...
}'

Since this starts with a single-quote, the first single quote it sees will terminate the string. The first single quote is around "${CI_PROJECT_ID}"', so the string being passed to curl is:

'{
  "id": '

To clarify why this curl command isn't throwing an exit code, this is still a legal string, and is perfectly fine to curl, so it sends it on to GitLab API. When parsing the data however, GitLab can't parse it, so nothing happens over there. It would result in a 4xx HTTP code however, but as you mentioned, you can't see some of that information in the job output.

To fix this, I'd remove the single quotes around $CI_PROJECT_ID:

'{
  "id": "${CI_PROJECT_ID}",
  ...
}'

or, since $CI_PROJECT_ID is and always will be an integer, you can remove all the quotes around it:

'{
  "id": $CI_PROJECT_ID,
  ...
}'

Generate GitLab Merge Request when a push comes in via CI/CD Pipeline using Bash and cURL

1 Answers1