Py4j launch_gateway not connecting properly

Question

I am trying to use py4j to open up a gateway that I can use to pass objects from java into python. When I try to open a gateway with the py4j function launch_gateway it does not seem to properly connect to my Java class. However, when I launch my java class in the command line and then connect to it in python using JavaGateway everything works as expected. I would like to be able to use the built in method as I am sure that I am not accounting for things that have already been considered in the design of py4j, but I'm just not sure what I'm doing wrong.

Let's say I wanted to create a gateway to the class sandbox.demo.solver.UtilityReporterEntryPoint.class. In the command line I can do this by executing the following:

java -cp /Users/grr/anaconda/share/py4j/py4j0.10.4.jar: sandbox.demo.solver.UtilityReporterEntryPoint py4j.GatewayServer

This launches as expected and I can use the methods in my class from within python after connecting to the gateway. So far so good.

My understanding of the py4j documentation would lead me to believe I should do the following to launch the gateway in python:

port = launch_gateway(classpath='sandbox.demo.solver.UtilityReporterEntryPoint')
params = GatewayParameters(port=port)
gateway= JavaGateway(gateway_parameters=params)

I get no errors when executing these three lines, but when I try to access my java class methods with gateway.entry_point.someMethod() it fails with the following error:

Py4JError: An error occurred while calling t.getReport. Trace: py4j.Py4JException: Target Object ID does not exist for this gateway :t at py4j.Gateway.invoke(Gateway.java:277) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:214) at java.lang.Thread.run(Thread.java:745)

Obviously something is not getting called correctly within launch_gateway or I am feeding it the wrong information.

In the py4j source code for launch_gateway you can see that given the inputs you provide and those constructed by the function, a command is constructed that eventually gets called by subprocess.Popen. So given the input passed to launch_gateway above the command passed into Popen would be:

command = ['java', '-classpath', '/Users/grr/anaconda/share/py4j/py4j0.10.4.jar:sandbox.demo.solver.UtilityReporterEntryPoint', 'py4j.GatewayServer', '0']

Passing this command to Popen returns the listening port as expected. However, connecting to this listening port still does not allow access to my class methods.

Finally, passing the command as a single string to Popen without the final argument ('0'), properly launches a gateway which again operates as expected. Having taken a glance at the Java source code for py4j.GatewayServer.class this makes no sense as the main method seems to indicate that the class should exit with status 1 if the length of arguments is 0.

At this point I'm kind of at a loss. I can hack my way into a workable solution, but as I said I'm sure that ignores important aspects of the gateway behavior and I don't like hacky solutions. I'd love to tag @Barthelemy in this one, but hopefully he reads this. Thanks in advance for any help.

EDIT

For now I have been able to work around this issue with the following steps.

1. Package entire project including all external dependencies into a single jar file magABM-all.jar, with 'Main-Class' set to UtilityReporterEntryPoint.

2. Include if...else block regarding presence of --die-on-exit exactly like it is in GatewayServer.java

3. Use subprocess.Popen to call the command to run the project jar.

UtilityReporterEntryPoint.java

public static void main(String[] args) throws IOException {
  GatewayServer server = new GatewayServer(new UtilityReporterEntryPoint());
  System.out.println("Gateway Server Started");
  server.start();
  if (args[0].equals("--die-on-exit")) {
    try {
        BufferedReader stdin = new BufferedReader(new InputStreamReader(System.in, Charset.forName("UTF-8")));
        stdin.readLine();
        System.exit(0);
    } catch (java.io.IOException e) {
        System.exit(1);
    }
  }
}

app.py

def setup_gateway()
    """Launch a py4j gateway using UtilityReporterEntryPoint."""
    process = subprocess.Popen('java -jar magABM-all.jar --die-on-exit', shell=True)
    time.sleep(0.5)
    gateway = JavaGateway()
    return gateway

In this way I can still use gateway.shutdown if necessary and if the python process that starts the py4j gateway dies or is closed the gateway will be closed.

N.B I would by no means consider this a final solution as py4j was written by much smarter individuals with a clear purpose in mind and I am sure that there is a way to manage this exact workflow within the confines of py4j. This is just a stopgap solution.

Answer 1

There are a few issues:

1. The classpath parameter in launch_gateway should be a directory or a jar file, not a class name. For example, if you want to include additional Java libraries, you would add them to the classpath parameter.

2. The error you receive when you call gateway.entry_point.someMethod() means that you have no entry point. When you call launch_gateway, the JVM is started with GatewayServer.main, which launches a GatewayServer with no entry point: GatewayServer server = new GatewayServer(null, port). It is not possible currently to use launch_gateway and specify an entry point.

3. When you start the JVM with java -cp /Users/grr/anaconda/share/py4j/py4j0.10.4.jar: sandbox.demo.solver.UtilityReporterEntryPoint py4j.GatewayServer I believe the JVM uses UtilityReporterEntryPoint as the main class. Although you did not provide the code, I assume that this class has a main method and that it launches a GatewayServer with an instance of UtilityReporterEntryPoint as the entry point. Note that there is a whitespace between the colon and the class name so UtilityReporterEntryPoint is seen as the main class and not as being part of the classpath.