How do stop Python running Py4J ClientServer

Question

I have a Java main() that is starting one or more Py4J ClientServer and python instance to connect back to the ClientServer (all use different port sets). This is working, but when the main completes, both Java and the python instance are not exiting

I'm starting the ClientServer as follows:

clientServer = new ClientServer(javaPort, GatewayServer.defaultAddress(), pythonPort,
    GatewayServer.defaultAddress(), GatewayServer.DEFAULT_CONNECT_TIMEOUT,
    GatewayServer.DEFAULT_READ_TIMEOUT, ServerSocketFactory.getDefault(), 
    SocketFactory.getDefault(), jep);

and python is started by passing in the two ports and doing as follows:

python_classifier.gateway = ClientServer(
    java_parameters=JavaParameters(port=javaPort, auto_convert=True, auto_close=True),
    python_parameters=PythonParameters(port=pythonPort, daemonize=False,
         daemonize_connections=True),
    python_server_entry_point=python_classifier)

I've tried setting deamonize=True, but then the connections start failing. I've also added a Java shutdown hook to try and kill the python Processes I've started, but it is not always being called.

So, any ideas on how to get python to quietly exit when the Java side has exited?

Both are reported issues. Java -> https://github.com/bartdag/py4j/issues/302. Python -> https://github.com/bartdag/py4j/issues/148 — David Wood, Dec 05 '19 at 22:23

David Wood · Answer 1 · 2019-05-28T16:42:16.227

I have lots of code around managing the python processes started from Java including shutdownHooks, finalizers() and timeout usage listeners. The main problem turned out to be that for a given ClientServer/Process pair, I was calling ClientServer.shutdown() before calling Process.destroyForcibly() After swapping the order and killing the process first , shutdown() no longer hangs in close() (see below).

There is a comment in the Py4J java code about the reader hanging in readLine() if the socket is not closed first. It still seems to be hanging despite this socket being closed first. The stack trace below is from Finalizer thread, but I got this in my own threads as well.

How do stop Python running Py4J ClientServer

1 Answers1