1

Added in May 1st:
I saw an issue about this error on the jira of apache-flink ,maybe it helps?

My system is CentOS7, python version 3.6.8, pyflink version 1.10.0;
I'm following this tutorial and trying to run a pyflink file; but I constantly get the error below;
I have tried to run it with options -pyarch and -pyexec venv.zip/venv/bin/python3 ,but it's useless.
I also add t_env.get_config().set_python_executable("python3") into the py-file but still the same error: 

[root@localhost pyflink]# flink run -m  localhost:8081  -pyarch venv.zip -pyexec venv.zip/venv/bin/python3  -py test_split_label.py 
Traceback (most recent call last):
  File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/root/wyl/test_file/pyflink/test_split_label.py", line 58, in <module>
    from pyflink.datastream import StreamExecutionEnvironment
  File "/tmp/pyflink/89713583-7229-4c30-93c9-4d543c360437/bd1f0746-4704-44e2-ba9e-3ef2bea03354pyflink.zip/pyflink/__init__.py", line 23, in <module>
RuntimeError: Python versions prior to 3.5 are not supported for PyFlink [sys.version_info(major=2, minor=7, micro=5, releaselevel='final', serial=0)].
org.apache.flink.client.program.OptimizerPlanEnvironment$ProgramAbortException
    at org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:87)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:321)
    at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:205)
    at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:138)
    at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:664)
    at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:213)
    at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:895)
    at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:968)
    at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
    at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:968)

my default python version has been set to Python3.6: 

[root@localhost pyflink]# python
Python 3.6.8 (default, Aug  7 2019, 17:28:10) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.version_info
sys.version_info(major=3, minor=6, micro=8, releaselevel='final', serial=0)

It seems that I should set some environmental variable ? but I don't know what name and path should I set. Now I have a $FLINK_HOME.
Any help is greatly appreciated.

Holly Wang
  • 11
  • 3
  • From your error traceback, it looks like flink is trying to use python 2.7, which is probably installed on your os as well. It may be that flink is installed in the wrong python env? Could you try checking the path Flink is installed under? – PirateNinjas Apr 28 '20 at 11:37
  • @PirateNinjas Thanks for your answer and I've checked the path, `[root@localhost pyflink]# pwd /usr/local/lib/python3.6/site-packages/pyflink ` The file works fine with the command `python xxx.py` or in pyflink-shell.sh. I'm really confused why it failed with `flink run -py` – Holly Wang Apr 29 '20 at 09:08
  • You could try setting your `PYTHONPATH` in the shell to be the correct python. Something like `export PYTHONPATH=` – PirateNinjas Apr 29 '20 at 10:03
  • @PirateNinjas same error TOT. I modify the `PYTHONPATH` like `export PYTHONPATH=/usr/bin/python3` , is it right? My soft link to `Python` like this `alias python=/usr/bin/python3` .But the error still raise. – Holly Wang Apr 29 '20 at 14:49
  • @snakecharmerb I have a line like `#!/usr/bin/env python3` in the head of the file. – Holly Wang May 06 '20 at 07:06

2 Answers2

1

it is because the python environment of the client side you are using is 2.7. What the api set_python_executable do is setting the python executable in cluster side. So if your PyFlink version is 1.10, you need to use virtualenv with 3.5+ python environment and you can refer to the doc https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/python/common_questions.html#execute-pyflink-jobs-with-python-virtual-environment. In PyFlink 1.11, you can directly config PYFLINK_CLIENT_EXECUTABLE to sepecify the python executable in client side.

Xingbo Huang
  • 363
  • 1
  • 5
0
./bin/flink run
      -pyclientexec venv.zip/venv/bin/python3 \
      -pyexec venv.zip/venv/bin/python3 \
      -py shipfiles/word_count.py

I followed by this tutorial, It's ok https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/cli/

HoangThang
  • 171
  • 1
  • 11