3

I'm new to GitLab. I am building my first pipeline to deploy the contents of my GitLab project to an FTP server with TLS encryption. I've written a Python script using ftplib to upload the files to the FTP server that works perfectly when I run it on my local Windows machine. The script uploads the full contents of the project to a folder on the FTP server. Now I'm trying to get it to work on GitLab by calling the script in the project's .gitlab-ci.yml file. Both the script and the yml file are in the top level of my GitLab project. The setup is extremely simple for the moment:

image: python:latest

deploy:
    stage: deploy
    script: 
        - python ftpupload.py
    only:
        - main

However, the upload always times out with the following error message:

  File "/usr/local/lib/python3.9/ftplib.py", line 156, in connect
    self.sock = socket.create_connection((self.host, self.port), self.timeout,
  File "/usr/local/lib/python3.9/socket.py", line 843, in create_connection
    raise err
  File "/usr/local/lib/python3.9/socket.py", line 831, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out
Cleaning up file based variables
ERROR: Job failed: exit code 1

Here's the basic setup for establishing the connection in the Python script that works fine locally but fails on GitLab:

class ReusedSslSocket(ssl.SSLSocket):
    def unwrap(self):
        pass


class MyFTP_TLS(ftplib.FTP_TLS):
    """Explicit FTPS, with shared TLS session"""
    def ntransfercmd(self, cmd, rest=None):
        conn, size = ftplib.FTP.ntransfercmd(self, cmd, rest)
        if self._prot_p:
            conn = self.context.wrap_socket(conn,
                                            server_hostname=self.host,
                                            session=self.sock.session)  # reuses TLS session            
            conn.__class__ = ReusedSslSocket  # we should not close reused ssl socket when file transfers finish
        return conn, size

session = MyFTP_TLS(server, username, password, timeout=None)
session.prot_p()

I know there are other tools like lftp and git-ftp that I could use in GitLab CI, but I've built a lot of custom functionality into the Python script and would like to use it. How can I successfully deploy the script within GitLab CI? Thanks in advance for your help!

torek
  • 448,244
  • 59
  • 642
  • 775
ionosphere
  • 373
  • 2
  • 13
  • 1
    Welcome to SO and GitLab! Is this running on GitLab.com, or your own self-managed GitLab? If it's GitLab.com, I'm pretty sure that the shared runners won't be able to open SFTP to arbitrary addresses - otherwise the service could be abused for mounting DoS/DDoS attacks. You would have to [install](https://docs.gitlab.com/runner/install/) and [register](https://docs.gitlab.com/runner/register/index.html) your own runner on a computer that can reach the target FTP server. – mike Oct 22 '21 at 03:37
  • Thanks, @mike! Excellent point, that makes sense. It’s my company’s self-managed GitLab instance. The runner executing the job was one of my organization’s shared runners used across various groups. Perhaps I need my own dedicated runner for my group? I’ll try that. Thanks! – ionosphere Oct 22 '21 at 04:33
  • 1
    Yes, try that. Since it's your own company's infrastructure, then hopefully you can control network restrictions (or make the case to the network admin's) between the runner computer and the FTP server. To aid in convincing them: there is no need to open up access between the GitLab server and the FTP server, only the runner will be connecting. – mike Oct 22 '21 at 08:23
  • Thanks, @mike! Creating a dedicated group runner like you suggested worked. The FTP connection now works perfectly with the group runner. Many thanks for your help! – ionosphere Oct 25 '21 at 20:47

1 Answers1

1

This requires that the GitLab Runner (which executes the pipeline) is able to make an SFTP connection to your FTP server.

Shared runners are likely locked down to only connect to the GitLab server (to prevent an attack vector).

To work around this, install your own runner and register it to your GitLab.

mike
  • 1,854
  • 17
  • 23