Info on our data flow pipeline we're referring to in this incident:
- pipeline is responsible for moving data from Oracle source to BigQuery;
- pipeline is written in Python3.6;
- it uses ojdbc, jdk and jaydebeapi;
- it is ensured in our code that all required libraries etc. are installed always on all the Data Flow workers before execution.
Problem description: 21/10 we experienced problem with Data Flow worker (in europe-west3 region) - see below log. It seems it couldn't load or use jaydebeapi library.
2020-10-21 17:28:42.792 CESTError message from worker: Traceback (most recent call last): File "apache_beam/runners/common.py", line 997, in apache_beam.runners.common.DoFnRunner._invoke_bundle_method File "apache_beam/runners/common.py", line 490, in apache_beam.runners.common.DoFnInvoker.invoke_start_bundle File "apache_beam/runners/common.py", line 496, in apache_beam.runners.common.DoFnInvoker.invoke_start_bundle File "/usr/local/lib/python3.7/site-packages/libs/dataflow/common.py", line 269, in start_bundle jars=[f"/tmp/{self.ojdbc_lib}"] File "/usr/local/lib/python3.7/site-packages/jaydebeapi/init.py", line 412, in connect jconn = _jdbc_connect(jclassname, url, driver_args, jars, libs) File "/usr/local/lib/python3.7/site-packages/jaydebeapi/init.py", line 199, in _jdbc_connect_jpype convertStrings=True) File "/usr/local/lib/python3.7/site-packages/jpype/_core.py", line 216, in startJVM ignoreUnrecognized, convertStrings, interrupt) SystemError: java.lang.ClassNotFoundException: org.jpype.classloader.DynamicClassLoader During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/dataflow_worker/batchworker.py", line 638, in do_work work_executor.execute() File "/usr/local/lib/python3.7/site-packages/dataflow_worker/executor.py", line 179, in execute op.start() File "apache_beam/runners/worker/operations.py", line 662, in apache_beam.runners.worker.operations.DoOperation.start File "apache_beam/runners/worker/operations.py", line 664, in apache_beam.runners.worker.operations.DoOperation.start File "apache_beam/runners/worker/operations.py", line 666, in apache_beam.runners.worker.operations.DoOperation.start File "apache_beam/runners/common.py", line 1014, in apache_beam.runners.common.DoFnRunner.start File "apache_beam/runners/common.py", line 999, in apache_beam.runners.common.DoFnRunner._invoke_bundle_method File "apache_beam/runners/common.py", line 1045, in apache_beam.runners.common.DoFnRunner._reraise_augmented File "/usr/local/lib/python3.7/site-packages/future/utils/init.py", line 446, in raise_with_traceback raise exc.with_traceback(traceback) File "apache_beam/runners/common.py", line 997, in apache_beam.runners.common.DoFnRunner._invoke_bundle_method File "apache_beam/runners/common.py", line 490, in apache_beam.runners.common.DoFnInvoker.invoke_start_bundle File "apache_beam/runners/common.py", line 496, in apache_beam.runners.common.DoFnInvoker.invoke_start_bundle File "/usr/local/lib/python3.7/site-packages/libs/dataflow/common.py", line 269, in start_bundle jars=[f"/tmp/{self.ojdbc_lib}"] File "/usr/local/lib/python3.7/site-packages/jaydebeapi/init.py", line 412, in connect jconn = _jdbc_connect(jclassname, url, driver_args, jars, libs) File "/usr/local/lib/python3.7/site-packages/jaydebeapi/init.py", line 199, in _jdbc_connect_jpype convertStrings=True) File "/usr/local/lib/python3.7/site-packages/jpype/_core.py", line 216, in startJVM ignoreUnrecognized, convertStrings, interrupt) SystemError: java.lang.ClassNotFoundException: org.jpype.classloader.DynamicClassLoader [while running 'Read from Oracle source/Read from database']
Problem occurred several times after running exactly same code again and then disappeared and everything worked well with the same code. Do you have any idea what could happen? It seems to us that it was something with infrastructure/worker provisioning etc.
