0

I need to check if a file exists in a gitlab deployment pipeline. How to do it efficiently and reliably?

sg_rs
  • 411
  • 3
  • 13

3 Answers3

1

Use gsutil ls gs://bucket/object-name and check the return value for 0.

If the object does not exist, the return value is 1.

John Hanley
  • 74,467
  • 6
  • 95
  • 159
1

You can add the following Shell script in a Gitlab job :

#!/usr/bin/env bash

set -o pipefail
set -u

gsutil -q stat gs://your_bucket/folder/your_file.csv

PATH_EXIST=$?
if [ ${PATH_EXIST} -eq 0 ]; then
  echo "Exist"
else
  echo "Not Exist"
fi

I used gcloud cli and gsutil with stat command with -q option.

In this case, if the file exists the command returns 0 otherwise 1.

Mazlum Tosun
  • 5,761
  • 1
  • 9
  • 23
  • This is exactly what I need. However, I have an upgrade for you script. Without pipefail and no global bash settings: 1. exit_code=0 2. gsutil stat gs://your_bucket/folder/your_file.csv || exit_code=$? – sg_rs Nov 24 '22 at 13:05
  • I tested without global bash settings and I have the same result for my script. – Mazlum Tosun Nov 24 '22 at 13:27
  • The difference is that in a Gitlab pipeline the gsutil stat will produce a non-zero result and teh pipeline will fail. You need to catch the error somehow. The || operator works – sg_rs Nov 24 '22 at 13:29
  • Ok I am going to test the script in a `Gitlab` job – Mazlum Tosun Nov 24 '22 at 13:31
1

This answer evolved from the answer of Mazlum Tosun. Because I think it is a substantial improvement with less lines and no global settings switching it needs to be a separate answer.

Ideally the answer would be something like this

- gsutil stat $BUCKET_PATH
- if [ $? -eq 0 ]; then  
  ... # do if file exists
  else
  ... # do if file does not exists
  fi

$? stores the exit_status of the previous command. 0 if success. This works fine in a local console. The problem with Gitlab will be that if the file does not exists, then "gsutil stat $BUCKET_PATH" will produce a non-zero exit code and the whole pipeline will stop at that line with an error. We need to catch the error, while still storing the exit code.

We will use the or operator || to suppress the error. FILE_EXISTS=false will only be executed if gsutil stat fails.

- gsutil stat $BUCKET_PATH || FILE_EXISTS=false
- if [ "$FILE_EXISTS" = false ]; then 
  ... # do stuff if file does not exist
  else
  ... # do stuff if file exists
  fi

Also we can use the -q flag to let the command stats be silent if that is desired.

- gsutil -q stat $BUCKET_PATH || FILE_EXISTS=false
sg_rs
  • 411
  • 3
  • 13