LFTP in gitlab CI: files are not updated on FTP server even if they are changed in the last commit

Question

we have an issue with GITLAB-CI pipelines when we commit a new modification on the server. We want to update the modification on the FTP server automatically.

To do that, we are using a simple CI configuration in gitlaci.yaml file:

deploy:
  script:
    - bash ./deploy.sh
  only:
    - develop
    - master

The content of deploy.sh is very simple too:

#!/bin/bash

echo "Deployment start";

echo "*** Target branch test:"
target="null"
case "$CI_COMMIT_REF_NAME" in
"develop")
    target=$DEV
    ;;
"master")
    target=$PROD
    ;;
*)
    echo "Pas de deployment sur cette branch"
    exit 1;
    ;;
esac
echo "*** $target"


echo "*** Install client FTP"
apt-get update -qq && apt-get install -y -qq lftp

echo "*** Copy file to target"
lftp -c "set ftp:ssl-allow no; open -u $USERNAME,$PASSWORD $HOST; mirror -Rnev ./ $target --ignore-time --parallel=10 --exclude-glob .git* --exclude .git/"

(the variables are will setup in CI variable, and most of the time it is working)

When we try to deploy changes by pipeline we have

sometimes he sends the files
sometimes don't send anything, zero files, even the ones changed in the last commit

(Reading database ... 29705 files and directories currently installed.)
Preparing to unpack .../lftp_4.7.4-1_amd64.deb ...
Unpacking lftp (4.7.4-1) ...
Setting up lftp (4.7.4-1) ...
*** Copy file to the target
Job succeeded

Any ideas?

The job is succeeded, but the commited file are not send on the FTP...
Which is a bit strange... How can I have a pipeline if no file is changed ^^ ??

Do you know what's happened?

Thank you,
Regards,
Nicolas

Additionnal log during my last issue. The related file updated in gitLab was trombi/index.php, it does not appear on the log... And on the server, it did not change at all:

 $ bash ./deploy.sh
 *** Déploiement de xxx - start:
 *** Target branche:
 *** ./www/xxxDev
 *** voir le resultat sur: xxx/Dev
 *** Install client FTP
 debconf: delaying package configuration, since apt-utils is not installed
 Selecting previously unselected package libtcl8.6:amd64.
 (Reading database ... 23968 files and directories currently installed.)
 Preparing to unpack .../libtcl8.6_8.6.9+dfsg-2_amd64.deb ...
 Unpacking libtcl8.6:amd64 (8.6.9+dfsg-2) ...
 Selecting previously unselected package tcl8.6.
 Preparing to unpack .../tcl8.6_8.6.9+dfsg-2_amd64.deb ...
 Unpacking tcl8.6 (8.6.9+dfsg-2) ...
 Selecting previously unselected package tcl-expect:amd64.
 Preparing to unpack .../tcl-expect_5.45.4-2_amd64.deb ...
 Unpacking tcl-expect:amd64 (5.45.4-2) ...
 Selecting previously unselected package expect.
 Preparing to unpack .../expect_5.45.4-2_amd64.deb ...
 Unpacking expect (5.45.4-2) ...
 Selecting previously unselected package lftp.
 Preparing to unpack .../lftp_4.8.4-2_amd64.deb ...
 Unpacking lftp (4.8.4-2) ...
 Setting up lftp (4.8.4-2) ...
 Setting up libtcl8.6:amd64 (8.6.9+dfsg-2) ...
 Setting up tcl8.6 (8.6.9+dfsg-2) ...
 Setting up tcl-expect:amd64 (5.45.4-2) ...
 Setting up expect (5.45.4-2) ...
 Processing triggers for mime-support (3.62) ...
 Processing triggers for hicolor-icon-theme (0.17-2) ...
 Processing triggers for libc-bin (2.28-10) ...
 *** Copy file to target
 Mirroring directory `.gitlab'
 Mirroring directory `adm'
 Mirroring directory `xxx'
 Mirroring directory `communaute'
 Mirroring directory `conso'
 Mirroring directory `cron'
 Mirroring directory `css'
 Mirroring directory `doc'
 Mirroring directory `eve'
 Mirroring directory `img'
 Finished mirror `img'
 Mirroring directory `js'
 Mirroring directory `ad/comi'
 Mirroring directory `comm/img'
 Finished mirror `conso'
 Mirroring directory `materiel'
 Finished mirror `js'
 Mirroring directory `monCompte'
 Mirroring directory `.gitlab/issue_templates'
 Finished mirror `association'
 Mirroring directory `money'
 Finished mirror `css'
 Mirroring directory `xxx'
 Mirroring directory `documents/parsedown-contents'
 Mirroring directory `events/championnats'
 Finished mirror `cron'
 Mirroring directory `proto'
 Finished mirror `adm/comite'
 Mirroring directory `adm/doc-legaux'
 Finished mirror `communaute/img'
 Mirroring directory `communaute/jou'
 Finished mirror `materiel'
 Mirroring directory `public'
 Finished mirror `.gitlab/issue_templates'
 Finished mirror `.gitlab'
 Mirroring directory `te'
 Finished mirror `monCompte'
 Mirroring directory `webServices'
 Finished mirror `xxx'
 Mirroring directory `admin/emails'
 Finished mirror `documents/parsedown-contents'
 Mirroring directory `documents/parsedown-master'
 Mirroring directory `money/cotisations'
 Finished mirror `events/championnats'
 Mirroring directory `events/festivals'
 Mirroring directory `proto/auth'
 Finished mirror `adm/do'
 Mirroring directory `adn/jeux'
 Finished mirror `communaute/jou'
 Mirroring directory `communaute/mem'
 Mirroring directory `public/noel'
 Mirroring directory `te/con'
 Finished mirror `adm/emails'
 Mirroring directory `adm/ppp'
 Finished mirror `events/fest'
 Mirroring directory `events/tour'
 Finished mirror `proto/auth'
 Finished mirror `proto'
 Mirroring directory `money/devis'
 Mirroring directory `communaute/membres/Ajouter'
 Finished mirror `te/cont'
 Finished mirror `te'
 Mirroring directory `communaute/membres/messageCollectif'
 Finished mirror `adm/popp'
 Mirroring directory `adm/reglement-interieur'
 Finished mirror `events/tour'
 Finished mirror `events'
 Mirroring directory `communaute/membres/trombi'
 Finished mirror `communaute/membres/Ajouter'
 Mirroring directory `communaute/membres/urgence'
 Finished mirror `communaute/membres/messageCollectif'
 Mirroring directory `communaute/outils'
 Finished mirror `adm/reglement-interieur'
 Mirroring directory `admin/roles-engagements-avantages'
 Finished mirror `communaute/membres/trombi'
 Mirroring directory `communaute/partenaires'
 Mirroring directory `public/noel/css'
 Finished mirror `communaute/membres/urgence'
 Finished mirror `communaute/membres'
 Mirroring directory `communaute/sponsors'
 Mirroring directory `communaute/outils/association'
 Finished mirror `admin/roles-engagements-avantages'
 Mirroring directory `admin/settings'
 Finished mirror `communaute/partenaires'
 Mirroring directory `communaute/outils/classements'
 Finished mirror `public/noel/css'
 Mirroring directory `public/noel/fonts'
 Finished mirror `communaute/sponsors'
 Mirroring directory `communaute/outils/smash-rules-sets'
 Finished mirror `communaute/outils/association'
 Mirroring directory `communaute/outils/teams'
 Finished mirror `admin/settings'
 Mirroring directory `public/noel/img'
 Finished mirror `communaute/outils/classements'
 Mirroring directory `public/noel/js'
 Mirroring directory `wwws/compte'
 Mirroring directory `money/cotisations/fiches'
 Finished mirror `admin/jeux'
 Finished mirror `documents/parsedown-master'
 Finished mirror `admin'
 Finished mirror `documents'
 Mirroring directory `money/img'
 Mirroring directory `money/note2Frais'
 Finished mirror `public/noel/fonts'
 Mirroring directory `money/tresorerie'
 Finished mirror `communaute/outils/smash-rules-sets'
 Mirroring directory `wwws/database'
 Finished mirror `communaute/outils/teams'
 Finished mirror `communaute/outils'
 Finished mirror `communaute'
 Mirroring directory `wwws/devis'
 Finished mirror `public/noel/img'
 Mirroring directory `wwws/discord'
 Finished mirror `public/noel/js'
 Finished mirror `public/noel'
 Finished mirror `public'
 Mirroring directory `wwws/email'
 Finished mirror `money/cotisations/fiches'
 Finished mirror `money/cotisations'
 Mirroring directory `wwws/emoji'
 Finished mirror `wwws/compte'
 Mirroring directory `wwws/files'
 Finished mirror `money/img'
 Mirroring directory `wwws/googleMaps'
 Mirroring directory `money/note2Frais/min'
 Finished mirror `money/tresorerie'
 Mirroring directory `money/note2Frais/dons'
 Finished mirror `wwws/devis'
 Mirroring directory `wwws/jou'
 Finished mirror `wwws/discord'
 Mirroring directory `wwws/membres'
 Finished mirror `money/devis'
 Mirroring directory `wwws/permissions'
 Finished mirror `wwws/dda'
 Mirroring directory `wwws/public'
 Finished mirror `wwws/email'
 Mirroring directory `wwws/resssppp'
 Finished mirror `wwws/emoji'
 Finished mirror `wwws/googleMaps'
 Finished mirror `wwws/files'
 Finished mirror `mon/note2Frais/aport'
 Finished mirror `mon/note2Frais/dons'
 Finished mirror `mon/note2Frais'
 Finished mirror `mon'
 Finished mirror `wwws/jou'
 Finished mirror `wwws/membres'
 Finished mirror `wwws/public'
 Finished mirror `wwws/permissions'
 Finished mirror `wwws/resetPasswords'
 Finished mirror `wwws'
 *** End
 ********* ATTENTION si pas de ligne entre 'End' et 'Copy file to target' alors il faut re lancer le déploiement
Running after script
00:02
Saving cache
00:02
Uploading artifacts for successful job
00:01
 Job succeeded

I confirm, the pipeline is running for a commit which has updated file:

Running with gitlab-runner 12.9.0-rc1 (a350f628)
   on docker-auto-scale ed2dce3a
Preparing the "docker+machine" executor
00:32
 Using Docker executor with image ruby:2.5 ...
 Pulling docker image ruby:2.5 ...
 Using docker image sha256:5a76bd71024b1c46b1e8871b205d025b03abe85ea for ruby:2.5 ...
Preparing environment
00:05
 Running on runner-ed2dce3a-project-10794060-concurrent-0 via runner-ed2dce3a-srm-1584724660-6891f84d...
Getting source from Git repository
00:03
 $ eval "$CI_PRE_CLONE_SCRIPT"
 Fetching changes...
 Initialized empty Git repository in /builds/xxx/prod/.git/
 Created fresh repository.
 From https://gitlab.com/xxx/prod
  * [new ref]         refs/pipelines/128249282 -> refs/pipelines/128249282
  * [new branch]      develop                  -> origin/develop
  * [new branch]      master                   -> origin/master
 Checking out 0f5997df as develop...

And the commit 0f5997df contains: Showing 1 changed file with 3 additions and 3 deletions

Maybe I can try to adapt my deploy.sh script to catch the answer and see if there is file copied... If no... Maybe I should retry... But if it does not work the first time, I doubt it will work a for the second time — NicoESIEA, Mar 20 '19 at 20:36
Pipeline example of no action: https://i.stack.imgur.com/TtdqW.png — NicoESIEA, Mar 22 '19 at 08:15
How does it work this FTP client ? How does it know which files are concerned??? Is there a way in git ci to see the last updated files?? — NicoESIEA, Mar 31 '19 at 07:10
I really need to understand what's happened... Does anybody met the same issue??? Question: Should I offer a bountie on this ticket? How can I? — NicoESIEA, Apr 06 '19 at 07:57
Again three times the same bug... I have to restart the process by adding a fake comment... I use as commit message always the same value: "Force redeploy" then I'm able to see how many times it does not work — NicoESIEA, Apr 13 '19 at 15:18
I will try with this information: https://stackoverflow.com/questions/55514613/how-can-i-know-the-updated-file-during-the-gitlabci-pipeline — NicoESIEA, Apr 18 '19 at 10:40
Does anybody know how can I get the log of my ci job ? I will try to parse this log to verify if the modified files are in the list... If they are not, then retry the entier job... — NicoESIEA, Oct 10 '19 at 13:56
Maybe I can catch the output of "lftp -c" command by redirect the std output (using the simple >filename) — NicoESIEA, Nov 25 '19 at 14:01
But... i would really like it to work, as expected… Is there really nobody else concerned by this question? — NicoESIEA, Mar 01 '20 at 08:28
If you are working with git repository, why don't you use git pull on your remote servers? — makozaki, Mar 02 '20 at 18:29
That is a great idea.. Thank you @makozaki But I have no ssh access to my server :( — NicoESIEA, Mar 02 '20 at 22:53
And another reason is, we are many poeple developping the app. We do not want give access to every body to the server. Using gitLab pipeline, there is no password to share, and the target can be hidden (using git variables) — NicoESIEA, Mar 02 '20 at 22:57
You could execute git pull with ssh command using pipeline and store private key in env so it wouldn't be visible, [example](https://docs.gitlab.com/ee/ci/ssh_keys/#ssh-keys-when-using-the-docker-executor). Yet still it wouldn't matter if you don't have ssh access to servers. — makozaki, Mar 03 '20 at 19:47
The solution proposed under this message does not work, today again I faced many time the issue.... I do not understand :( HELP — NicoESIEA, Apr 15 '20 at 12:51
Still the same bug... I'm more and more often obliged to force redeploy by changing litle stuff in my code... Is there some other idea to help me?? — NicoESIEA, Aug 05 '20 at 10:00

score 2 · Accepted Answer · answered Oct 30 '20 at 17:31

At the end, I finaly decide to move my projet from gitlab to github. The CI/CD is pretty the same, except the need/possibility to split the yaml files into each target env.

This is what I did and one file for dev, an other for prod. The contents of the two files are pretty the same, I show you the content of DEV file, the only diff is for URL_DEV secret value

on:
  push:
    branches:
      - develop
name: Publish on https://www.mydomain.fr/myprojectname
jobs:
  FTP-Deploy-Action:
    name: FTP-Deploy-Action
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2.1.0
      with:
        fetch-depth: 2
    - name: FTP-Deploy-Action
      uses: SamKirkland/FTP-Deploy-Action@3.1.1
      with:
        ftp-server: ${{ secrets.FTP_HOSTNAME }}/${{ secrets.URL_DEV }}/
        ftp-username: ${{ secrets.FTP_USERNAME }}
        ftp-password: ${{ secrets.FTP_PASSWORD }}

All the secrets variables are the same as I had in gitlab;

By doing this change, I solved the issue related to this ticket + I removed the limitation due to gitlab shared runner minute

ByeBye GitLab, Hello GitHub...And thank to everybody who help me to find a fix

makozaki · Answer 2 · 2020-03-05T20:11:24.500

1

I would suggest adding set -euo pipefail at the beggining of deploy.sh so pipeline wouldn't ignore script errors. You said that status is always Success regardless of command result.

Looks like others had problem with missing output from LFTP, there is a suggestion to use unbuffer to solve this problem. Note that this might require installation of expect package.

You could also try increasing verbosity level -v/-vv/-vvv to see what really went wrong.

UPDATE: My sugesstion was to install expect package witha apt-get and use unbuffer command together with lftp to make sure all logs are flushed to console. You could try updating deploy.sh install line and copying files line as follows:

echo "*** Install client FTP and expect"
apt-get update -qq && apt-get install -y -qq lftp expect

echo "*** Copy file to target and unbuffer lftp output"
unbuffer lftp -c "<your command>"

Regarding verbosity I'd start with mirror command:

#You could start with level one verbosity and increase it if needed
lftp -c "... mirror -v ..."
lftp -c "... mirror -vv ..."
lftp -c "... mirror -vvv ..."

edited Mar 05 '20 at 20:11

answered Mar 04 '20 at 07:39

makozaki

3,772
4
23
47

Thank you very much for you answer makozaki. I did the set up for pipefail... we will see. I prefer continu with LFTP first...and search for another tool later. May I ask you how to increase verbosity level (-v/-vv/-vvv) to see what really went wrong? Thank you – NicoESIEA Mar 05 '20 at 16:05
I meant to use `unbuffer` together with `lftp` to make sure logs are flushed and visible. Verbosity can be used for many commands. I'd start with `mirror`, from [man page](https://lftp.yar.ru/lftp-man.html). I updated my answer with clarification. Hope it helps. – makozaki Mar 05 '20 at 20:13
Thank you for the UPDATE. So, the command I use becomes: unbuffer lftp -c "set ftp:ssl-allow no; open -u $USERNAME,$PASSWORD $HOST; mirror -Rnev ./ $target --ignore-time --parallel=10 --exclude-glob .git* --exclude .git/" – NicoESIEA Mar 06 '20 at 08:53
Including mirror, is it correct to have: unbuffer lftp -c "set ftp:ssl-allow no; open -u $USERNAME,$PASSWORD $HOST; mirror -vvv -Rnev ./ $target --ignore-time --parallel=10 --exclude-glob .git* --exclude .git/" ? – NicoESIEA Mar 06 '20 at 08:55
Yes, that's what I meant. Can you just give it a try? – makozaki Mar 06 '20 at 09:11
The modification trigger a new deployment... and it was working as expected... now let wait until a new random error and we will see if the additionnal can help ;) Thank you – NicoESIEA Mar 06 '20 at 13:49
I faced the same issue (commit and push, but the latest modification is not deployed) I will add the additionnal log into my question (don't worry, if needed I can add another bounty on this question, by the way, really big thank you for your help) – NicoESIEA Mar 09 '20 at 20:46
As far as I can see, there is no more bug... maybe the use of unbuffer fixe the issue... I will wait 10 more days and we will see... Is my bounty still valide if I mark my question as solved in 10 days? – NicoESIEA Mar 12 '20 at 21:13
I did many many test and it look to be ok using unbuffer. How can I restore my bounty to validate your answer? Should I simply validate your answer - even if I validate after the end of the bounty periode? – NicoESIEA Mar 15 '20 at 14:02
Don't worry about the bounty if my answer helped then you can just accept it :) – makozaki Mar 18 '20 at 18:10
I unvalidate the answer because I faced the same issue as before... But this time I got more logs. The updated file was /trombi/index.php but the ftp client only test (mirror ) the folder ( Mirroring directory `communaute/membres/trombi', then Finished without processing any file)... It looks like there is still an issue – NicoESIEA Mar 20 '20 at 18:10
In order to defined if there is something processed during the job, I search the text ' ( in the log... In my exemple, there is nothing :( – NicoESIEA Mar 20 '20 at 18:12
So command finished without error, there is log that folder you wanted to mirror has been processed. Yet still you claim a file that should be there is missing. How about you check logs at the beginning to see which commit has been checked out. There should be a line: `Checking out as ...`. Can you checkout locally this commit to make sure the file you claim to be missing from remote server is there? – makozaki Mar 20 '20 at 21:32
I updated the question description to add the details. **Yes** the commit well contains the update file (**Showing 1 changed file with 3 additions and 3 deletions**). NOTE: what I'm doing when I observed this kind of bug, I just add a fake modification in the file and commit/push to trigger another pipeline... And it works.... – NicoESIEA Mar 21 '20 at 08:58
Is it possible the link between lftp and git is bugged? Does anybody knows how it works? – NicoESIEA Mar 21 '20 at 18:16
I think this whole question is to vague and misleading. It is not a GitLab problem as it executes pipeline and return success due to non failing lftp command. I would try to reproduce it from other than CI machine. From your logs we see that everything is working fine from GitLab pov. Maybe investigating your firewalls or lftp server will bring you closer to fixing the problem. For starters I would try reading [here](https://www.ncftp.com/ncftpd/doc/misc/ftp_and_firewalls.html#problems) – makozaki Mar 21 '20 at 20:21
Thank you for your time and investigation. I do not use any firewall. I only go to GitLab web site and use the IDE tool. Same thing for LFTP, I only install the client using apt-get, Agree with you, the issue should be at LFTP side... But what can I do? Should I rename my question for "LFTP in gitlab CI" ? – NicoESIEA Mar 22 '20 at 09:17
I tried the same code in another git project, with another gitlab ci pipeline (more simple, with only one master branch) - same result... with the same random happening – NicoESIEA Mar 28 '20 at 19:32
The solution proposed does not work, today again I faced many time the issue.... I do not understand :( HELP – NicoESIEA Apr 15 '20 at 12:51
Try getting access via ssh and using git commands to update git repository on remote server – makozaki Apr 15 '20 at 14:09
You already proposed this solution Makozaki, and I have already said I have no ssh access to my server... Really I can not :( – NicoESIEA Apr 16 '20 at 15:29
Still the same error, more and more often... This solution does not work :( Does anybody has another idea? – NicoESIEA Aug 05 '20 at 09:59
I add a new comment to explain what I do for the moment... But it is still bugged... I have just a workaround :( – NicoESIEA Sep 22 '20 at 16:12
Just a rhetorical questions here: Who administrates this remote server you cannot ssh to? Do you have any other options, rights for this server? Maybe you can achieve what you are currently doing in other way? What is the purpose of your FTP server? Is this some kind of Wordpress app? Other way could be to ask administrator to setup gitlab worker on this machine and give it only access rights to some directory which needs to be updated. That way you could execute updates via gitlab pipeline directly on this machine. Donno much about your problem but seems like lftp is fundamentally bad solutio – makozaki Sep 23 '20 at 08:47
This is lower-cost service from OVH. The SSH access is not activated with this offer :( – NicoESIEA Oct 21 '20 at 09:22

NicoESIEA · Answer 3 · 2020-09-22T23:16:52.037

The error is still present For the moment, I found only one workaround, using this code to warn me in case of need:

echo "*** Install client FTP"
apt-get update -qq && apt-get install -y -qq lftp expect

echo "*** Copy file to target"
unbuffer lftp -c "set ftp:ssl-allow no; open -u $USERNAME,$PASSWORD $HOST; mirror -vvv -Rnev ./ $target --ignore-time --parallel=10 --exclude-glob .git* --exclude .git/" >> log_${CI_JOB_ID}_lftp.txt

cat log_${CI_JOB_ID}_lftp.txt

count=$(grep -c "' (" log_${CI_JOB_ID}_lftp.txt)
echo $count

# lesser than 2: log file + at least one file
if [ $count -lt 2 ]
then
    echo "erreur: need to FORCE REDEPLOY"
    echo "erreur: need to FORCE REDEPLOY"
    echo "erreur: need to FORCE REDEPLOY"
    echo "erreur: need to FORCE REDEPLOY"
    exit 1;
else
    echo "End ***"
fi

I test if there are updated file, in case of no updated file (count < 2, count=1 or 0) exit 1; in order to force the user to redeploy...
But the developper still must to create a new commit, not only restart the pipline.

Why -lt 2 and not 1? Because the log file will be added on the server too. So I need to check there is at least 2 files send, and not only 1.
(Do not worry, the previous log file will be removed because the mirror will not see it in the source code)

score 1 · Answer 4 · answered Oct 22 '20 at 07:32

If this is a lower-cost ftp service I'd expect it to be unstable and unreliable. For this to work best you could try reducing file transfer to minimum.

After some research I came across git-ftp tool which seems to serve your purpose best - deploy to ftp server after each commit only changed files.

Might be worth a try. There are even preinstalled docker images.

deploy_production:
  image: dotsunited/git-ftp
  stage: deploy
  only:
    - master
  script:
    - git ftp push -v --user $FTP_DEPLOY_USER --passwd $FTP_DEPLOY_PASSWORD $FTP_DEPLOY_HOST

LFTP in gitlab CI: files are not updated on FTP server even if they are changed in the last commit

4 Answers4

Linked