Questions tagged [py4j]

Py4J enables Python programs to dynamically access arbitrary Java objects

Py4J enables Python programs running in a Python interpreter to dynamically access Java objects in a Java Virtual Machine. Methods are called as if the Java objects resided in the Python interpreter and Java collections can be accessed through standard Python collection methods. Py4J also enables Java programs to call back Python objects. Py4J is distributed under the BSD license.

Here is a brief example of what you can do with Py4J. The following Python program creates a java.util.Random instance from a JVM and calls some of its methods. It also accesses a custom Java class, AdditionApplication to add the generated numbers.

 from py4j.java_gateway import JavaGateway

 gateway = JavaGateway()                   # connect to the JVM

 random = gateway.jvm.java.util.Random()   # create a java.util.Random instance

 number1 = random.nextInt(10)              # call the Random.nextInt method

 number2 = random.nextInt(10)

 print(number1,number2)

(2, 7)

 addition_app = gateway.entry_point        # get the AdditionApplication instance

 addition_app.addition(number1,number2)    # call the addition method

9
235 questions
3
votes
1 answer

JavaPackage object is not callable error: Pyspark

Operations like dataframe.show() , sQLContext.read.json works fine , but most functions gives "JavaPackage object is not callable error" . eg : when i do dataFrame.withColumn(field_name, monotonically_increasing_id()) I get an error File…
Himaprasoon
  • 2,609
  • 3
  • 25
  • 46
3
votes
2 answers

What is the entry point for its py4j gatewayServer in spark?

I was trying to run a java function in pyspark using py4j. Py4j enables accessing java objects in a JVM. I created another instance of a JVM and was able run the java function successfully. py4j enables this communication via GatewayServer…
Sanjay Kumar
  • 31
  • 1
  • 5
3
votes
2 answers

Connect to JVM for debugging that has been spawned by a Python script

I have a Python code, which I run with IntelliJ. The Python code will execute a java command, which will eventually spawn a JVM. The Python and JVM communicates using Py4J and custom sockets. How can I connect to the JVM for debugging purposes using…
Dyin
  • 5,815
  • 8
  • 44
  • 69
3
votes
0 answers

PY4J with generics

With the following class definition: class MyClass[T : ClassTag]{ def this(stuff : T){ // .... } } PY4J will throw the following error: py4j.protocol.Py4JError: An error occurred while calling None.MyClass. Trace: py4j.Py4JException:…
Dyin
  • 5,815
  • 8
  • 44
  • 69
3
votes
0 answers

Zeppelin: Constructor org.apache.spark.api.python.PythonRDD does not exist

IPython notebook Started as per docs (PYSPARK_DRIVER_PYTHON=ipython PYSPARK_DRIVER_PYTHON_OPTS="notebook" ./bin/pyspark), then filled in with: from os import path from tempfile import gettempdir #from pyspark import SparkFiles filename =…
A T
  • 13,008
  • 21
  • 97
  • 158
3
votes
2 answers

What happens to imports when a new process is spawned?

What happens to imported modules variables when a new process is spawned? IE with concurrent.futures.ProcessPoolExecutor(max_workers=settings.MAX_PROCESSES) as executor: for stuff in executor.map(foo, paths): where: def foo(str): x =…
user2757902
  • 493
  • 7
  • 18
3
votes
0 answers

Tracing down missing import from py4j on error "trying to call a package"

Given the following python snippet: from pyspark.mydb.mydb import * class HBaseTest: def __init__(self): from pyspark.context import SparkContext sc = SparkContext('local[4]', 'PythonTest') self._hbctx = MyDbSQLContext(sc) The line…
WestCoastProjects
  • 58,982
  • 91
  • 316
  • 560
2
votes
1 answer

Creating sparkContext on Google Colab gives: `RuntimeError: Java gateway process exited before sending its port number`

Following are the dependencies, which got installed successfully. !apt-get install openjdk-8-jre !apt-get install scala !pip install py4j !wget -q https://downloads.apache.org/spark/spark-2.4.8/spark-2.4.8-bin-hadoop2.7.tgz !tar xf…
Hira Tanveer
  • 155
  • 1
  • 4
  • 14
2
votes
0 answers

Cant handle Py4JJavaError

Trying to handle a Py4JJavaError that is raised when entering invalid sql code to spark.sql(). My function is as follows: import py4j def sql_to_df_and_create_view(sql_string: str, view_name: str): ''' Takes in a SQL command in string format…
Kal2c
  • 21
  • 2
2
votes
1 answer

Is it possible to call Java methods on a different server from Python using Py4J

I am using py4j for communication between python and java. Currently, the java processes and python are in same server. I would like to know is it possible to separate these two into different servers without much changes in the current approach…
Remis Haroon - رامز
  • 3,304
  • 4
  • 34
  • 62
2
votes
0 answers

How to import Scala case objects from jvm?

Example : object x { case object obj1 case object obj2 } I am trying to use a scala function that takes as variables a case object(for example obj1 as shown above). I am doing that because i want to a call scala object using py4j from…
2
votes
0 answers

py4j - use the java member object in multiprocessing

I have two java objects that take some input and return an output. To make them available in Python I start a JVM using subprocess.popen(java -Xmx3g -Xms3g -jar ) I then build a gateway using: gateway = JavaGateway( …
Abhishek Jain
  • 568
  • 2
  • 4
  • 15
2
votes
0 answers

Run a pyspark script as part of a Spark-scala application

My usecase goes like this: Read one or more dataframes in a spark-scala app and register them as tables. Get a python callable which would run pyspark based transformations on these dataframes. Register the transformed dataframes as tables into the…
Ankit Khettry
  • 997
  • 1
  • 13
  • 33
2
votes
0 answers

py4j.protocol.Py4JError: An error occurred while calling o66.createStream- Amazon kinesis,pyspark

I am trying to use spark streaming in python to read data from Amazon Kinesis Data Stream using KinesisUtils package and I get an error. from pyspark import SparkContext from pyspark.streaming import StreamingContext from pyspark.streaming.kinesis…
Misha B
  • 21
  • 1
2
votes
0 answers

Use python to connect to Java GUI read and control the Interface

Hi I'm fairly new to this but I thought I would try and ask if there is a way for python3.7 to read information from a java based UI. I want to control center buttons, read values and configure settings. I was hoping it would do this via some sort…
DTpy
  • 21
  • 2