3

I am using the free shared runners on the gitlab.com environment. I have a gitlab-CI pipeline that runs the following lftp commands one after the other:

  • lftp -c "set ftp:ssl-allow no; open -u $USERNAME,$PASSWORD $HOST; glob -a rm -r ./httpdocs/*"
  • lftp -c "set ftp:ssl-allow no; open -u $USERNAME,$PASSWORD $HOST; mirror -R public/ httpdocs --ignore-time --parallel=50 --exclude-glob .git* --exclude .git/"

The purpose of these commands is to delete the content of the httpdocs folder (previous files) and then upload new build artifact.

The CI pipeline is triggered from a CMS. It sometimes happen that the content editors update the content in parallel, resulting in a lot of triggers that run in parallel (the pipeline takes about 3 minutes to finish).

The pipeline will then start failing with the following error:

rm: Access failed: 550 /httpdocs/build-html-styles.css: No such file or directory

This happens because a file deleted by another pipeline is queued for deletion. A very similar error happens when the httpdocs folder is completely empty. This results in my whole pipeline failing (the second upload lftp command does not get executed at all).

Examples of failing pipelines and their output:

How do I prevent this from happening? Using lftp to upload the artifact is not a must - I am running the node:8.10.0 docker image. Gitlab-ci.yml file in question.

Miha Šušteršič
  • 9,742
  • 25
  • 92
  • 163
  • Well, we can't see your CI/CD pipelines. You believe this happens because two pipelines run at the same? So can you inspect your CI/CD pipelines outputs to see if there were two running at the same time? Maybe just implement a simple file locking with polling waiting? – KamilCuk Dec 14 '19 at 00:03
  • 1
    @KamilCuk - sorry about that, I made the pipeline outputs public now. The jobs I am referencing in the question were started on Dec 11, for example #102227849, #102227848, #102227856. I also managed to reproduce the issue today by triggering jobs running in parallel. I do not know what you mean by file locking with polling waiting though - is this done on the FTP server or using lftp? – Miha Šušteršič Dec 14 '19 at 09:43
  • 1
    I only see `404 Not found` for all the links. – KamilCuk Dec 14 '19 at 14:21

3 Answers3

1

I was commenting about simple file locking with simple active polling waiting. I have no experience with lftp, but scrambling from various internet resources like this, I have written the following. I see that lftp does not support file locking in the protocol, so you could something like this:

const="set ftp:ssl-allow no; open -u $USERNAME,$PASSWORD $HOST"
# wait until file exists
while lftp -c "$const; df lockfile"; do sleep 1; done
# create the lockfile
lftp -c "$const; mkdir lockfile"
# the work
lftp -c "$const; glob -a rm -r ./httpdocs/*"
lftp -c "$const; mirror -R public/ httpdocs  --ignore-time --parallel=50 --exclude-glob .git* --exclude .git/"
# remove lockfile
lftp -c "$const; rmdir lockfile"

I used mkdir and rmdir and a directory instead of a file, because I don't know how to create an empty file with lftp. There is still a race condition between finding file and creating it, but it should protect at least against two concurrent accesses. To protect more you could do something like sleep 0.$(printf "%02d" $((RANDOM / 10))) - make the sleep time random, so they enter creating a file less "concurently".

Also just in case, I wouldn't mirror to httpdocs directory, but to some temporary directory like tmp=httpdocs_$(uuidgen); lftp "mirror .. $tmp" that could be later renamed lftp 'rmdir httpdocs; rename $tmp httpdocs", for making deployments safer with less downtime (less time with httpdocs beeing empty). For future I suggest to just move to a safer/more advanced protocol of connecting with your remote server, that supports file locking. Like ssh. Or maybe samba.

KamilCuk
  • 120,984
  • 8
  • 59
  • 111
  • Thanks a lot for this - mirroring to a temporary directory did the trick. Also appreciate the samba and ssh suggestions, but unfortunately I am not able to change the server configuration (provided by a third party) – Miha Šušteršič Dec 18 '19 at 16:25
0

lavv17/lftp issue 302 advocated for lftp to skip such content, but that has not gained any traction.

The mailing list suggests

To delete a directory use rmdir

In your case: rm -rf httpdocs/ (followed by mkdir httpdocs if you need the empty folder)

VonC
  • 1,262,500
  • 529
  • 4,410
  • 5,250
  • I tried changing my command to `lftp -c "set ftp:ssl-allow no; open -u $USERNAME,$PASSWORD $HOST; glob -a rm -rf httpdocs && mkdir httpdocs`. The script is still failing at this command when two of them are running in parallel, but the exit code is now: `ERROR: Job failed: exit code 1`. You can check jobs #379392355 and #379392354 for reference. – Miha Šušteršič Dec 14 '19 at 09:56
0

Concurrency level of 50 with --parallel=50 seems high. I had a 550 permission error as well using lftp and lowering the level of concurrency fixed the issue for me.

Timothy Gonzalez
  • 663
  • 7
  • 9