Looping through files in ftp using ftplib returns "No such file or directory" in second loop

Question

I am trying to loop through the files on the ftp and then store them. However, on the second iteration, I receive the error:

FileNotFoundError: [Errno 2] No such file or directory:

Here is my code:

# TODO: auth
from ftplib import FTP

def extract(environment):

    ftp = FTP(auth["host"])

    # Monitor and extract
    with ftp.login(user=auth['username'], passwd=auth['password']) as ftp:

        folders = []
        try:
            folders = ftp.nlst()
        except:
            print('Probably no folders in this directory')

        for f in folders:

            # Go into subfolder per subfund
            path = "".join(['/',f])
            ftp.cwd(path)

            # List files
            files = []
            try:
                files = ftp.nlst()
            except:
                print('Probably no files in this directory')

            for filename in files:
                if ".csv" in filename:

                    with open(filename, 'r+') as source_file:

                        print('opened, this works for the 1st only')
                        store_to_gcs(source_file, filename)


def store_to_gcs(source_file, filename)
    # TODO: bucket = storage.bucket(app=app)
    # After it I store it to GCS, does it have anything to do with it?

    storage_ref = "test/" + filename
    blob = bucket.blob(storage_ref)
    blob.upload_from_file(source_file)

The with open(filename, 'r+') as source_file works only for the first file in files, but not for the second.

I can confirm that I am in the right directory as I did ftp.pwd() to confirm.

So if you put a breakpoint right before the `with open` line, what is the value of `filename` when it fails, and how does it differ from the first time around? — Random Davis, Feb 19 '19 at 19:46
1st: 'ffpos1_708524_57474156_18022019_036521_1.csv', 2nd: 'fflia1_708470_57474842_18022019_036521_1.csv' — WJA, Feb 19 '19 at 19:48
@MartinPrikryl Yes you need the outer loop as the idea is to go through folders on the FTP and then within each folder extract all .csv files. I added it there as I am suspecting that it has something to do with the problem. — WJA, Feb 19 '19 at 19:50
Correct. But it works in the first round. How else do you open it as a remote file using ftplib? — WJA, Feb 19 '19 at 19:52
Ah you are right, totally missed that! What do you suggest to replace the with open with? — WJA, Feb 19 '19 at 19:53

Martin Prikryl · Accepted Answer · 2019-02-19T20:19:08.690

open(filename, 'r+') opens a local file. While I believe you want to open a remote file.

You probably have a local copy of ffpos1_708524_57474156_18022019_036521_1.csv, but not of fflia1_708470_57474842_18022019_036521_1.csv. That would explain why open seemingly succeeds for the first iteration.

There's no open-like function in ftplib.

There are two solutions:

Download the file to memory - i.e. to BytesIO file-like object.
See Retrieve data from gz file on FTP server without writing it locally.

You can then pass the BytesIO to blob.upload_from_file.

This is easy to implement, but can be a problem, if the file is too large.
```
for filename in files:
    if ".csv" in filename:
        flo = BytesIO()
        ftp.retrbinary('RETR ' + filename, flo.write)

        flo.seek(0)
        store_to_gcs(flo, filename)
```
Implement a custom file-like object that can read the remote file on the fly, as needed. That's more complicated, but possible.
See Get files names inside a zip file on FTP server without downloading whole archive.

Using the first example I still get that the file was not found in memory. — WJA, Feb 19 '19 at 20:03

Looping through files in ftp using ftplib returns "No such file or directory" in second loop

1 Answers1