Questions tagged [py4j]

Py4J enables Python programs to dynamically access arbitrary Java objects

Py4J enables Python programs running in a Python interpreter to dynamically access Java objects in a Java Virtual Machine. Methods are called as if the Java objects resided in the Python interpreter and Java collections can be accessed through standard Python collection methods. Py4J also enables Java programs to call back Python objects. Py4J is distributed under the BSD license.

Here is a brief example of what you can do with Py4J. The following Python program creates a java.util.Random instance from a JVM and calls some of its methods. It also accesses a custom Java class, AdditionApplication to add the generated numbers.

 from py4j.java_gateway import JavaGateway

 gateway = JavaGateway()                   # connect to the JVM

 random = gateway.jvm.java.util.Random()   # create a java.util.Random instance

 number1 = random.nextInt(10)              # call the Random.nextInt method

 number2 = random.nextInt(10)

 print(number1,number2)

(2, 7)

 addition_app = gateway.entry_point        # get the AdditionApplication instance

 addition_app.addition(number1,number2)    # call the addition method

9
235 questions
2
votes
0 answers

How to get original Python error from a Py4JJavaError raised in a PySpark UDF

I am using PySpark UDFs to execute code on a Spark worker. If an exception is raised in the UDF this is wrapped in a Py4JJavaError and re-raised in Python. In order to process the error correctly I need the original error. Is there a way to get it…
eega
  • 479
  • 8
  • 20
2
votes
1 answer

Py4j Cannot connect to Java Server

I was trying to write a simple program to setup a connection between python and java using py4j. I wrote the following two lines hoping that everything would run since I'm not making any changes from py4j.java_gateway import JavaGateway,…
Clock Slave
  • 7,627
  • 15
  • 68
  • 109
2
votes
0 answers

pyspark: error IllegalArgumentException while calling fit function on word2vec

i'm testing this spark script on my local machine (ubuntu on windows) AND on a hortonworks HDP sandbox. The result is the same. This is what it looks like: from pyspark.sql import SparkSession from pyspark.sql.functions import split from…
2
votes
0 answers

Call getNextException from pyspark

I am trying to write data to db2 via pyspark and want to get better error messages from failures. I know I can reach java errors like this: from py4j.protocol import Py4JJavaError try: data_frame.write.jdbc('jdbc...', table='some_table',…
user3124181
  • 782
  • 9
  • 24
2
votes
0 answers

Pyspark throwing error: py4j.Py4JException: Method __getstate__([]) does not exist

I have a method in Scala file that returns a py4j.java_collections.JavaMap in my pyspark code. I believe that i can use dictionary methods on my Java map. But, I am not able to. I am trying to use a user defined function from my pyspark code where i…
robinhood
  • 21
  • 1
  • 2
2
votes
2 answers

PySpark NoSuchMethodError: sun.nio.ch.DirectBuffer.cleaner when inserting data into DB

I am receiving this error while trying to insert a large dataframe into Postgres. NoSuchMethodError: sun.nio.ch.DirectBuffer.cleaner Here is a full error. There are a lot of operations before, so there are no reason in attaching them to the…
Anton Bondar
  • 319
  • 2
  • 4
  • 13
2
votes
0 answers

'JavaPackage' object is not callable pyspark 2.3.0 Anaconda Win10

I am starting with pySpark. I have installed it in anadonda in Win10. I have copied an example and when I execute the code, I am getting this error: Traceback (most recent call last): File ".\testingSpark.py", line 7, in spark =…
Manwelanza
  • 31
  • 4
2
votes
0 answers

How can I finish pyspark unit tests gracefully withou `py4j.java_gateway:Error`?

How can I finish pyspark unit tests gracefully without see this annoying error when the test ends? It seems like the python client is losing communication with the spark context or something like that. ERROR:py4j.java_gateway:Error while sending…
Hanan Shteingart
  • 8,480
  • 10
  • 53
  • 66
2
votes
1 answer

Importing py4j using Eclipse

I've read the first part of the py4j.org introduction, then I skipped down to the Eclipse section. I installed the Eclipse plugins found here: http://eclipse.py4j.org/ and restarted Eclipse afterwards. I have a class in the pre-existing Java project…
Kyle N Payne
  • 79
  • 11
2
votes
1 answer

Using py4j to send matrices to from Python to Java as int[][] arrays

I've been using py4j to build a user-friendly Python library around a less user-friendly Java library. For the most part, this has been a breeze, and py4j has been a great tool. However, I've come across a snag when sending matrices between Python…
nben
  • 646
  • 5
  • 20
2
votes
0 answers

Interaction between custom Java code and Python code in Spark

We've got Java libraries for doing certain things (querying the backend intelligently, mostly) in a Spark environment. I'd like to use their functionality but develop in Python. Is there a standard way of doing this (`this' probably meaning:…
Johannes Bauer
  • 462
  • 5
  • 15
2
votes
0 answers

py4j.protocol.Py4JNetworkError: An error occurred while trying to connect to the Java server

I m trying to understand this code which uses Py4j. However each time I run the code I'm getting the same error. I have py4j installed in my Ubuntu 14.04. The jar file is in usr/share/py4j. The code is from nltk.tokenize.punkt import…
shiju
  • 31
  • 3
2
votes
1 answer

Py4Java: ImportError: No module named numpy when running Python shell for Apache Spark

I'm attempting to follow the live coding in this Apache Spark talk Here is my IPython notebook, up to the point where I encounter the error: so numpy is installed: ~ $ pip install numpy Requirement already satisfied (use --upgrade to upgrade):…
Micah Stubbs
  • 1,827
  • 21
  • 34
2
votes
1 answer

How to view imported classes from py4j gateway

Consider the following py4j gateway init code: from py4j.java_gateway import java_import, JavaGateway, GatewayClient gateway = JavaGateway(GatewayClient(port=gateway_port),…
WestCoastProjects
  • 58,982
  • 91
  • 316
  • 560
2
votes
1 answer

how is scala type casting done in py4j?

Is there a way to do type casting like it is done is scala/java when using py4j to trigger jvm? Basically, I would like to translate this: someOtherTypeInstance.asInstanceOf[RDD[Any]] Into something like: someOtherTypeInstance =…
Winston Chen
  • 6,799
  • 12
  • 52
  • 81