Questions tagged [py4j]

Py4J enables Python programs to dynamically access arbitrary Java objects

Py4J enables Python programs running in a Python interpreter to dynamically access Java objects in a Java Virtual Machine. Methods are called as if the Java objects resided in the Python interpreter and Java collections can be accessed through standard Python collection methods. Py4J also enables Java programs to call back Python objects. Py4J is distributed under the BSD license.

Here is a brief example of what you can do with Py4J. The following Python program creates a java.util.Random instance from a JVM and calls some of its methods. It also accesses a custom Java class, AdditionApplication to add the generated numbers.

 from py4j.java_gateway import JavaGateway

 gateway = JavaGateway()                   # connect to the JVM

 random = gateway.jvm.java.util.Random()   # create a java.util.Random instance

 number1 = random.nextInt(10)              # call the Random.nextInt method

 number2 = random.nextInt(10)

 print(number1,number2)

(2, 7)

 addition_app = gateway.entry_point        # get the AdditionApplication instance

 addition_app.addition(number1,number2)    # call the addition method

9
235 questions
-1
votes
1 answer

pyspark got error Py4JJavaError when using SparkContext.wholeTextFiles method

I am trying to use SparkContext.wholeTextFiles for reading some log files in a directory, and got error with the below config: OS: Windows 10 Python version: 3.8.8 pyspark version: 3.1.2 java jdk: 1.8.0_91 hadoop version: 3.2.2 spark :…
Sina
  • 431
  • 4
  • 7
-1
votes
1 answer

How to createDataFrame in pyspark?

Dear Spark users / creators, I am trying to create a Spark DataFrame according to the documentation: https://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.SparkSession.Builder When I execute the following according to the…
-1
votes
1 answer

How to use NLTK in Jython, or PY4J cross platforn?

I have two questions, this could go into different directions based on opinion, but as of right now I am writing a Java/Kotlin API. So far it is compatible across all platforms, excluding IOS(have not tested). A few of the tasks it runs is python…
Corie LeClair
  • 51
  • 1
  • 9
-1
votes
1 answer

What is wrong with appName function?

When I run the spark's main function in jupyter python I get an error at .appName function spark=ps.sql.SparkSession.builder\ .master('local[4]')\ .appName('spark-lecture')\ .getOrCreate() sc= spark.sparkContext Py4JError:…
-1
votes
1 answer

Pyspark Py4j IllegalArgumentException with spark.createDataFrame and pyspark.ml.clustering

Let me disclose the full background of my problem first, I'll have a simplified MWE that recreates the same issue at the bottom. Feel free to skip me rambling about my setup and go straight to the last section. The Actors in my Original Problem: A…
Lukas Thaler
  • 2,672
  • 5
  • 15
  • 31
-1
votes
1 answer

pySpark 1.6 can't execute Java code via py4j despite it works in pySpark 2.0

Could somebody suggest a possible workaround except updating the Spark version? I wasn't able to spot a root-cause with enabled debug logging in spark and in python. Steps to reproduce GIVEN a spark 1.6.3 installation…
-1
votes
1 answer

createDataFrame (pyspark) generates a weird error (py4j error)

I wrote these simple 4 lines of code: import pyspark from pyspark.sql import SparkSession spa = SparkSession.builder.getOrCreate() spa.createDataFrame([(1,2,3)], ["count"]) but that createDataFrame function is generating this huge…
DobleR
  • 83
  • 1
  • 6
-1
votes
1 answer

How to pass complex Java objects instances to Python script

I have such Java object: public class MyObject { private Integer counter = 0; public void increment() { this.counter++; } // Getters, setters } In main() method i am calling method increment(): public class Main { …
EazyH
  • 57
  • 6
-2
votes
1 answer

Spark cannot resolve column

While running below code getting the error..it is azure Data bricks hands on EDA. df_typed = spark.sql("SELECT cast(Price as int), cast(Age as int), cast(KM as int), FuelType, cast(HP as int), cast(MetColor as int), cast(Automatic as…
-3
votes
1 answer

How to run a java program(.java code file) from a python program?

I basically want to write a python code, from where I send an input string to jave program, I want the java program to execute it and return the result to the python file. Is it possible?
User5421
  • 1
  • 2
1 2 3
15
16