For local execution and testing, I'd like to show more log messages in my Python Flink application. However, setting the logging level has no effect and more importantly no logs that happen during actual execution of the graph are shown. Note that this is only a problem when executing the script directly, not when say using the K8s operator for which I can override logConfiguration/log4j-console.properties to manipulate the level and message format.
Here's a MWE:
from logging import DEBUG, getLogger
from pyflink.common import Types
from pyflink.datastream import StreamExecutionEnvironment
logger = getLogger(__name__)
logger.setLevel(DEBUG)
print("PRINT is visible")
logger.info("INFO should be visible but is not")
logger.warning("WARNING is visible")
def test_operator(n: int) -> int:
print("PRINT is visible")
logger.info("INFO should be visible but is not")
logger.warning("WARNING is not visible either")
return n
env = StreamExecutionEnvironment.get_execution_environment()
source = env.from_collection(range(5), type_info=Types.INT())
source.map(test_operator).print()
env.execute()
Running this with python test.py produces
- all the print messages
- the first warning (top level, not in operator)
- none of the info messages
As an additional debugging step, I saw the various logging configuration templates in Flink's install directory, and they all had a logging level of DEBUG.
How can I configure my environment to show the logs?
