0

On CentOS (CentOS 8.0.1905 (64bit)) I tried to run Python(3.6.8) Version of WordCount program on Flink(1.9) as described here. I got error as below. The same environment works fine with Java version of WordCount program. What is that I am missing here? Any help is greatly appreciated. Error

$ ./bin/flink run -py examples/python/table/batch/word_count.py
Starting execution of program
org.apache.flink.client.program.OptimizerPlanEnvironment$ProgramAbortException
    at org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:83)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:576)
    at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:438)
    at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:274)
    at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:746)
    at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:273)
    at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:205)
    at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1010)
    at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1083)
    at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
    at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1083)
Hellen
  • 3,472
  • 5
  • 18
  • 25
Sarma
  • 11
  • 4

3 Answers3

0

I think the information you provided does not show root cause. Generally speaking，we have two ways to run the example in Flink 1.9, one is the answer 1 provided earlier, which is to install the PyFlink from source, and the other is to download the binary package. as follows:

  1. Check the Python python --version It’s better to use Python 2.7.6+ for Flink 1.9.

  2. Download curl -O http://mirrors.gigenet.com/apache/flink/flink-1.9.1/flink-1.9.1-bin-scala_2.11.tgz tar -zxvf flink-1.9.1-bin-scala_2.11.tgz

  3. Config(Optional) vi flink-1.9.1/conf/flink-conf.yaml -> rest.port: 4000

  4. Start cluster cd flink-1.9.1; bin/start-cluster.sh

  5. Run word count example ./bin/flink run -py examples/python/table/batch/word_count.py

More detail can be found here: https://enjoyment.cool/2020/01/19/Three-Min-Series-Run-the-Example-of-WordCount-in-PyFlink-1-9/#more

Hoping it is helpful to you!

jincheng sun
  • 1
  • 1
  • 1
    The site you link to appears to be your own. Please review the [promotion guideline.](/help/promotion) – tripleee Jan 20 '20 at 08:13
0

Thanks to Jincheng Sun for your encouragement and support

I moved $FLINK_HOME/opt/python/__MACOSX to one level above. Program ran without any errors. As I am running on Linux, I do not need __MACOSX. I will simply delete it

Sarma
  • 11
  • 4
-1

Before you run the test you should build the source and install the PyFlink by using pip, detail can be found here: https://ci.apache.org/projects/flink/flink-docs-release-1.9/flinkDev/building.html#build-pyflink.

jincheng sun
  • 1
  • 1
  • 1
    It looks more like comment then an answer. Please update your answer according to [how to answer](https://stackoverflow.com/help/how-to-answer) guide – Anna Jan 17 '20 at 04:40
  • I did not install Flink from source. Instead I downloaded. From log I could see "2020-01-17 10:39:40,271 ERROR org.apache.flink.client.python.PythonEnvUtils - Create symbol link for pyflink lib failed." Message I set PYTHONPATH as below PYTHONPATH=$FLINK_HOME/opt/python and in this directory unzipped pyflink directory I tried setting PYTHONPATH=$FLINK_HOME/opt/python/pyflink error flink/flink-python/src/main/java/org/apache/flink/client/python/PythonDriverEnvUtils.java – Sarma Jan 17 '20 at 07:22
  • Please share the log file: log/flink-xxx-client-xxxdeMacBook-Pro.local.log due to the information you provided does not show root cause. Or please have look at my answer 2 as well. – jincheng sun Jan 19 '20 at 04:36
  • Hi, Sarma, I do not have permission to comment on your answer, so comment here on your answer "I moved $ FLINK_HOME / opt / python / __ MACOSX to one level above. Program ran without any errors. As I am running on Linux, I do not need __MACOSX. I will simply delete it " :) I'm curious why you encountered this problem. Normally you don't need to decompress any zip file, and you won't encounter `__MACOSX`. – jincheng sun Jan 21 '20 at 04:54
  • You are correct. Do not unzip files in $FLINK_HOME/opt/python directory – Sarma Jan 22 '20 at 10:27