Questions tagged [py4j]

Py4J enables Python programs to dynamically access arbitrary Java objects

Py4J enables Python programs running in a Python interpreter to dynamically access Java objects in a Java Virtual Machine. Methods are called as if the Java objects resided in the Python interpreter and Java collections can be accessed through standard Python collection methods. Py4J also enables Java programs to call back Python objects. Py4J is distributed under the BSD license.

Here is a brief example of what you can do with Py4J. The following Python program creates a java.util.Random instance from a JVM and calls some of its methods. It also accesses a custom Java class, AdditionApplication to add the generated numbers.

 from py4j.java_gateway import JavaGateway

 gateway = JavaGateway()                   # connect to the JVM

 random = gateway.jvm.java.util.Random()   # create a java.util.Random instance

 number1 = random.nextInt(10)              # call the Random.nextInt method

 number2 = random.nextInt(10)

 print(number1,number2)

(2, 7)

 addition_app = gateway.entry_point        # get the AdditionApplication instance

 addition_app.addition(number1,number2)    # call the addition method

9
235 questions
5
votes
1 answer

py4j.protocol.Py4JNetworkError : An error occurred while trying to connect to the Java server

I have the following simple example from the py4j document: from py4j.java_gateway import JavaGateway def main(): print("Hello") gateway = JavaGateway() # connect to the JVM random = gateway.jvm.java.util.Random() #…
Edamame
  • 23,718
  • 73
  • 186
  • 320
5
votes
1 answer

java.util.HashMap missing in PySpark session

I'm working with Apache Spark 1.4.0 on Windows 7 x64 with Java 1.8.0_45 x64 and Python 2.7.10 x86 in IPython 3.2.0 I am attempting to write a DataFrame-based program in an IPython notebook that reads from and writes back to an SQL Server…
Techrocket9
  • 2,026
  • 3
  • 22
  • 33
4
votes
1 answer

UnsupportedOperationException while streaming data through pyspark

I am using this simple piece of code to read a stream of json files from a directory. The code works just fine on Databricks notebook however throws me an error while running it locally. I am using databricks-connect (version 8.1) to connect and run…
jainit
  • 41
  • 2
4
votes
1 answer

pyspark - Error while loading .csv file from url to Spark

pyspark load data from url url = "https://github.com/jokecamp/FootballData/blob/master/openFootballData/cities.csv" from pyspark import SparkFiles spark.sparkContext.addFile(url) spark.read.csv(SparkFiles.get("cities.csv"), header=True) However,…
4
votes
1 answer

How to fix Py4JJavaError: An error occurred while calling collectToPython

I'm trying to use pyspark interpreter on a zeppelin notebook deployed using Kubernetes. I have configured spark to use spark executors as well (5 cores, 1G storage). However, when I try to run pandas/seaborn and manipulate pandas dataframe, I get…
Joshua Villanueva
  • 179
  • 1
  • 5
  • 13
4
votes
1 answer

Convert a list in Scala to Python list or dataFrame

I have a 2d list in Scala called dataList and I want to convert it to a Pandas DataFrame. val dataList: List[List[Int]] = tempData.toList If I want to print dataList, everything works fine and the type of object in Python is
MTT
  • 5,113
  • 7
  • 35
  • 61
4
votes
4 answers

pyspark: call a custom java function from pyspark. Do I need Java_Gateway?

I wrote the following MyPythonGateway.java so that I can call my custom java class from Python: public class MyPythonGateway { public String findMyNum(String input) { return MyUtiltity.parse(input).getMyNum(); } public static…
Edamame
  • 23,718
  • 73
  • 186
  • 320
4
votes
1 answer

How to get SparkContext from JavaSparkContext in PySpark?

When I run PySpark, executing sc._gateway.help(sc._jsc) successfully gives me some nice output like JavaSparkContext extends org.apache.spark.api.java.JavaSparkContextVarargsWorkaround implements java.io.Closeable { | | Methods defined here: | …
Uri Laserson
  • 2,391
  • 5
  • 30
  • 39
3
votes
1 answer

Converting Scala case class to PySpark schema

Given a simple Scala case class like this: package com.foo.storage.schema case class Person(name: String, age: Int) it's possible to create a Spark schema from a case class as follows: import org.apache.spark.sql._ import…
RvdV
  • 406
  • 3
  • 10
3
votes
1 answer

IllegalArgumentException: 'Unsupported class file major version 55'

I was doing the assignment of "advanced machine learning and signal processing" in Coursera. I get the encountered with this error "Py4JavaEror". This is the first assignment of this course. It was said to be done in IBM Watson studio but doing it…
Rabin ghimire
  • 35
  • 1
  • 4
3
votes
1 answer

How do stop Python running Py4J ClientServer

I have a Java main() that is starting one or more Py4J ClientServer and python instance to connect back to the ClientServer (all use different port sets). This is working, but when the main completes, both Java and the python instance are not…
David Wood
  • 434
  • 1
  • 5
  • 14
3
votes
1 answer

Py4J callback from Java Runnable

I'm currently trying to do the following using Py4J: Define a method ("executor") in Python which calls JVM methods Define a Python ("callback") object implementing a JVM interface Construct a JVM object given this callback object Call a method on…
Macuyiko
  • 85
  • 1
  • 10
3
votes
1 answer

IllegalArgumentException: u'Wrong FS: file://spark-warehouse, expected: file:///'

I am trying to load my Postgres database into Spark using PySpark: from pyspark import SparkContext from pyspark import SparkConf from random import random #spark conf conf =…
FullStack
  • 5,902
  • 4
  • 43
  • 77
3
votes
2 answers

Different / better approaches for calling python function from Java

I am quite new to python and am trying to call python's function from java. My primary requirements are these: call should be transparent, in the sense that it should not require modifying .py file simply to enable it to be called from java. I…
Mahesha999
  • 22,693
  • 29
  • 116
  • 189
3
votes
2 answers

Fast conversion of Java array to NumPy array (Py4J)

There are some nice examples how to convert NumPy array to Java array, but not vice versa - how to convert data from Java object back to NumPy array. I have a Python script like this: from py4j.java_gateway import JavaGateway gateway =…
msi_gerva
  • 2,021
  • 3
  • 22
  • 28
1 2
3
15 16