Open a file (of type BytesIO) with the function

Question

I have the following pieces of code:

this part generates the CSV in memory:

def to_csv(events: list) -> io.BytesIO():
    if not events:
        return None
    bio = io.BytesIO()
    iow = io.TextIOWrapper(bio)
    writer = csv.DictWriter(iow, fieldnames=events[0].keys())
    writer.writeheader()
    writer.writerows(events)
    iow.flush()
    bio.seek(0)
    return bio

this part sends this file to the FTP server:

def send_data(self, bytes: io.BytesIO) -> str:
    filename = f"{time.time()}.csv"
    if not bytes:
        self.__logger.warning("No data to send")
        return None
    try:
        self.__ftp.storbinary(f"STOR {filename}", bytes)
    except ftp.all_errors as e:
        self.__logger.error(
            "FTP fail data send",
            extra={
                "host": self.__cfg.ftp.host,
                "type": type(e).__name__,
                "line": e.__traceback__.tb_lineno,
                "file": __file__,
                "detail": e,
            },
        )
        return None

The file that is passed to send_data is closed. How do I reopen it? I tried to do something like this:

f = open(bytes, "rb")
print(f.getvalue())

But an error is returned:

expected str, bytes or os.PathLike object, not _io.BytesIO

Traceback:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/operators/python.py", line 171, in execute
    return_value = self.execute_callable()
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/operators/python.py", line 189, in execute_callable
    return self.python_callable(*self.op_args, **self.op_kwargs)
  File "/opt/airflow/dags/accounting/main.py", line 29, in process_events
    result = Uploader(logger, cfg).send_data(csv_file)
  File "/opt/airflow/dags/accounting/packages/uploader.py", line 57, in send_data
    self.__ftp.storbinary(f"STOR {filename}", bytes)
  File "/usr/local/lib/python3.7/ftplib.py", line 513, in storbinary
    buf = fp.read(blocksize)
ValueError: I/O operation on closed file.

`storbinary` does not close the file passed to it. Nothing in this code would close the file. — user2357112, Aug 05 '22 at 12:17
please include the full traceback - it will show where the error is coming from — ti7, Aug 05 '22 at 12:24
@user2357112 If you write the following at the beginning of the `send_data` function: `print(bool(bytes.closed))`, then the console outputs `True` — Romsik788, Aug 05 '22 at 12:24
@Romsik788: Wait, at the *beginning* of `send_data`? Are you saying you're trying to *pass* a closed BytesIO object to this function? That's hopeless. Your data is already gone. Don't close the BytesIO instance in the first place. — user2357112, Aug 05 '22 at 12:27
@user2357112 then how do I return the file from the `to_csv` function so that file doesn't close? Since I don't close it — Romsik788, Aug 05 '22 at 12:30
Oh. I see it now. The `io.TextIOWrapper` is closing the wrapped object when the wrapper gets reclaimed. I'm not sure there's a good way to avoid that without either keeping the wrapper alive, or not using TextIOWrapper. — user2357112, Aug 05 '22 at 12:34

score -1 · Accepted Answer · answered Aug 05 '22 at 12:37

-1

Once to_csv is done using the io.TextIOWrapper, it needs to detach the wrapper from the underlying io.BytesIO object, to stop the wrapper from trying to close the BytesIO object:

iow.detach()

answered Aug 05 '22 at 12:37

user2357112

260,549
28
431
505

Open a file (of type BytesIO) with the function

1 Answers1