Questions tagged [py4j]

Py4J enables Python programs to dynamically access arbitrary Java objects

Py4J enables Python programs running in a Python interpreter to dynamically access Java objects in a Java Virtual Machine. Methods are called as if the Java objects resided in the Python interpreter and Java collections can be accessed through standard Python collection methods. Py4J also enables Java programs to call back Python objects. Py4J is distributed under the BSD license.

Here is a brief example of what you can do with Py4J. The following Python program creates a java.util.Random instance from a JVM and calls some of its methods. It also accesses a custom Java class, AdditionApplication to add the generated numbers.

 from py4j.java_gateway import JavaGateway

 gateway = JavaGateway()                   # connect to the JVM

 random = gateway.jvm.java.util.Random()   # create a java.util.Random instance

 number1 = random.nextInt(10)              # call the Random.nextInt method

 number2 = random.nextInt(10)

 print(number1,number2)

(2, 7)

 addition_app = gateway.entry_point        # get the AdditionApplication instance

 addition_app.addition(number1,number2)    # call the addition method

9
235 questions
0
votes
0 answers

ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host. Solution to this?

I am getting this error whenever I try to fit my model using als in pyspark. Can anybody help with this? Exception happened during processing of request from ('127.0.0.1', 55552) Traceback (most recent call last): File…
0
votes
1 answer

How can I instantiate a Scala case object in Python using Py4J?

I have a Scala case object defined like so: object DurationUnitsOfMeasure { sealed abstract class DurationUnitOfMeasure(val name : String) { override def toString : String = name lazy val initial: Char = name.charAt(2).toLower } case…
jamiet
  • 10,501
  • 14
  • 80
  • 159
0
votes
1 answer

How can I convert a Scala HashSet to a Python set using py4j?

I have an object in Scala that returns a Set[String]: object MyObject { val setOfStrings = Set[String]("string1","string2") } I can successfully refer to that val from my Python code (using Py4J) with…
jamiet
  • 10,501
  • 14
  • 80
  • 159
0
votes
2 answers

Py4JError: An error occured while calling o129.and. Trace: py4j.Py4JException: Method and ([class java.lang.string]) does not exist

I am trying to check for a condition in pyspark dataframe and add values to a column like below: DF: cd id Location A A A A AA A A AAA A B B A BB B A BBB Expected output: cd id Location A …
Padfoot123
  • 1,057
  • 3
  • 24
  • 43
0
votes
1 answer

Py4JError while converting csv file to parquet using jupyter-notebook

I want to convert a csv to parquet file using jupyter notebook, python3. However, i get the next error: Py4JJavaError Traceback (most recent call last) Py4JJavaError: An error occurred while calling o40.parquet. :…
jusmin
  • 7
  • 2
0
votes
1 answer

(PySpark) StringIndexer Error: py4j.protocol.Py4JJavaError: An error occurred while calling o46.fit

I have a dataFrame in PySpark. I want to use StringIndexer for my label column, so I defined a function as: def indexer(column, dataframe): from pyspark.ml.feature import StringIndexer # Indexing the column stringIndexer =…
Saeid SOHEILY KHAH
  • 747
  • 3
  • 10
  • 23
0
votes
1 answer

Can't install py4jdbc

I have to install py4jdbc module in python on windows 10, but i am having the followings errors when i try to use pip install py4jdbc or from the source: Error 1: "The executable sbt cannot be found" Pip Error Error 2: "the file doesn't…
Nacho
  • 3
  • 5
0
votes
1 answer

How to test jdbc connection in python?

I tried using the py4j referred Connecting and testing a JDBC driver from Python from py4j.JavaGateway import java_gateway # Open JVM interface with the JDBC Jar jdbc_jar_path = 'C:\Program Files\CData\CData JDBC Driver for MongoDB…
0
votes
1 answer

Pass python functions to Scala RDD in pyspark

I have a scala library which (to put it simple) receives a function, applies it to an RDD and returns another RDD def runFunction(rdd: RDD, function: Any => Any) = { .... val res = rdd.map(function) ... } In scala the usage would be…
alexlipa
  • 1,131
  • 2
  • 12
  • 27
0
votes
0 answers

Fix for Py4JJavaError while trying to convert 1.7 lac size pyspark df to pandas df

> Py4JJavaError: An error occurred while calling o342.collectToPython. : > org.apache.spark.SparkException: Job aborted due to stage failure: > Task 36 in stage 14.0 failed 1 times, most recent failure: Lost task > 36.0 in stage 14.0 (TID 675,…
Nikita Rathi
  • 101
  • 1
  • 6
0
votes
1 answer

Py4JError: An error occurred while calling o25.isBarrier. Trace: py4j.Py4JException: Method isBarrier([]) does not exist

I am getting this error while using spark : Py4JError: An error occurred while calling o25.isBarrier. Trace: py4j.Py4JException: Method isBarrier([]) does not exist version check and configuraion setup cross checked from pyspark import…
0
votes
0 answers

Is it possible that a py4j JavaObject get garbage collected while calling its java method?

From the py4j official blog (yes it's an ancient blog), the following quoted sentence doesn't seem to be reasonable to me. if javaObject is no longer referenced in the Python program, javaObject could be garbage collected before method1() is…
0
votes
1 answer

How do you consume a scala.collection.mutable.ArrayBuffer over py4j

I'm attempintg to consume a SparkListenerJobStart object in Python, but the py4j bindings do not understand ArrayBuffers (or, I guess, scala.collection.Seqs in general). What do?
badp
  • 11,409
  • 3
  • 61
  • 89
0
votes
0 answers

py4j from pyspark - launch_gateway() error

Step1 - I have created a jar out of sample java code- MyPythonGateway.jar Java Class- import py4j.GatewayServer; public class MyPythonGateway{ public int findMyNum(String input){ return(1); } public static void main(String[]…
0
votes
1 answer

Access scala function in PySpark

I have a Scala library which contains some utility codes and UDF for the Scala Spark API. However, I would love to now start to use this Scala library with PySpark. Using Java based classes seems to work pretty OK like outlined Running custom Java…
Georg Heiler
  • 16,916
  • 36
  • 162
  • 292