How to convert DataFrame to Dataset in Apache Spark in Java?

Question

I can convert DataFrame to Dataset in Scala very easy:

case class Person(name:String, age:Long)
val df = ctx.read.json("/tmp/persons.json")
val ds = df.as[Person]
ds.printSchema

but in Java version I don't know how to convert Dataframe to Dataset? Any Idea?

my effort is:

DataFrame df = ctx.read().json(logFile);
Encoder<Person> encoder = new Encoder<>();
Dataset<Person> ds = new Dataset<Person>(ctx,df.logicalPlan(),encoder);
ds.printSchema();

but the compiler say:

Error:(23, 27) java: org.apache.spark.sql.Encoder is abstract; cannot be instantiated

Edited(Solution):

solution based on @Leet-Falcon answers:

DataFrame df = ctx.read().json(logFile);
Encoder<Person> encoder = Encoders.bean(Person.class);
Dataset<Person> ds = new Dataset<Person>(ctx, df.logicalPlan(), encoder);

Seems like the API is missing for Java /w Spark 1.6 – Leet-Falcon Jan 07 '16 at 11:43 — Leet-Falcon, Jan 07 '16 at 11:43
how can I write the missing api in Java? – Milad Khajavi Jan 07 '16 at 11:47 — Milad Khajavi, Jan 07 '16 at 11:47

score 14 · Accepted Answer · answered Jan 07 '16 at 12:03

Official Spark docs suggest in Dataset API the following:

Java Encoders are specified by calling static methods on Encoders.

List<String> data = Arrays.asList("abc", "abc", "xyz");
Dataset<String> ds = context.createDataset(data, Encoders.STRING());

Encoders can be composed into tuples:

Encoder<Tuple2<Integer, String>> encoder2 = Encoders.tuple(Encoders.INT(), Encoders.STRING());
List<Tuple2<Integer, String>> data2 = Arrays.asList(new scala.Tuple2(1, "a");
Dataset<Tuple2<Integer, String>> ds2 = context.createDataset(data2, encoder2);

Or constructed from Java Beans by Encoders#bean:

Encoders.bean(MyClass.class);

Is there a performance implication when transforming a DataFrame to a Dataset? Specifically, what happens when I attach an Encoder to a DataFrame? Is the encoder 'lazy' (i.e. it does nothing unless a typed operation is invoked) or does it have to process the DataFrame first? — Marsellus Wallace, Apr 27 '17 at 19:19

score 6 · Answer 2 · answered Oct 27 '16 at 06:15

6

If you want to convert a generic DF to a Dataset in Java, you can use RowEncoder class like below

DataFrame df = sql.read().json(sc.parallelize(ImmutableList.of(
            "{\"id\": 0, \"phoneNumber\": 109, \"zip\": \"94102\"}"
    )));

    Dataset<Row> dataset = df.as(RowEncoder$.MODULE$.apply(df.schema()));

answered Oct 27 '16 at 06:15

nomad

5,442
1
18
8

Like this in scala ```val encoder = Encoders.product[Foo] df.as[Foo](encoder)``` – Painy James Apr 11 '17 at 13:07
Problem is DataFrame class is gone from Spark 2! – Thomas Decaux Mar 01 '18 at 10:34
Its not working as is. Dataset dataset = df.as(RowEncoder$.MODULE$.apply(df.schema())); getting an error " incompatible type". – Vamshavardhan Reddy Apr 12 '18 at 06:23

How to convert DataFrame to Dataset in Apache Spark in Java?

Edited(Solution):

2 Answers2

Linked