import local packages inside PyFlink

Question

I'm trying to write a local package in PyFlink project. But I can only import via relative path.

like

from .package import func

Can I use absolute paths in packages inside PyFlink project imported as env.add_python_file('/path_to_project') ?

korolkevichm · Accepted Answer · 2021-12-22T12:54:16.700

For using absolute paths answer from https://lists.apache.org/list.html?user@flink.apache.org: full answer here for abstract structure the directory:

flink_app/
    data_service/
        filesystem.py
    validator/
        validator.py
    common/
        constants.py
    main.py <- entry job

When submitting the PyFlink job you can specify python files and entry main module with option --pyFiles and --pyModule1, like:

$ ./bin/flink run --pyModule flink_app.main --pyFiles ${WORKSPACE}/flink_app

In this way, all files under the directory will be added to the PYTHONPAHT of both the local client and the remote python UDF worker.

1 Answers1