0

I am writing a dataflow job using Apache-Beam which requires to import data in BigQuery using the FTPS [from ftplib import FTP_TLS] server. But the moment I try to import FTPS Class [ftps = FTP_TLS('ftp.xxxxx.xxx')], I get the error as "TypeError: can't pickle SSLContext objects". I try to put the package in setup.py, to import ftplib package, but still the same error persists.

Does Apache-Beam not support data from FTPS server and if it supports then how to resolve the error?

Mikhail Berlyant
  • 165,386
  • 8
  • 154
  • 230
nikhil sahai
  • 531
  • 1
  • 5
  • 4

1 Answers1

0

I'm not sure because you didn't provide a code snippet. But i think that you create your FTP_TLS object in the init. All objects that are created in the init must be serializable.

You can try to create the object in the Process function. For more details see: see Ankur answer on this issue

More info see the documentation:

https://beam.apache.org/documentation/programming-guide/#core-beam-transforms

https://beam.apache.org/documentation/programming-guide/#requirements-for-writing-user-code-for-beam-transforms

Geddy Schellevis
  • 1
  • 2