Spark: JavaRDD to JavaPairRDD<>

Question

I have a JavaRDD<Tuple2<String, String>> and need to transform it to JavaPairRDD<String, String>. Currently I am doing it by simply writing map function that just returns the input tuple as is. But I wonder if there is a better way?

Might be.. I also finally found PairJavaRDD.fromJavaRDD(rdd) — YuliaSh., Nov 19 '14 at 19:15
if you are the one constructing the `JavaRDD>` e.g., from a map transformation of a JavaPairRDD, you could instead call mapToPair and avoid having a JavaRDD in the first place. — vefthym, Feb 21 '17 at 13:30

YuliaSh. · Accepted Answer · 2015-03-05T07:35:39.867

14

JavaPairRDD.fromJavaRDD(rdd) is one of solutions

edited Mar 05 '15 at 07:35

answered Nov 19 '14 at 19:22

YuliaSh.

795
1
6
23

JavaRDD buildingRDD = jsc.sparkContext().parallelize(listSmartBuilding); I want to iterate over this JavaRDD, could you help me. SmartBuildingNew is a POJO class.jsc is the JavaStreamingContext object – Anshul Kalra Feb 11 '16 at 02:59

score 4 · Answer 2 · answered Jun 17 '15 at 13:39

4

For reverse conversion, this seems to work:

JavaRDD.fromRDD(JavaPairRDD.toRDD(rdd), rdd.classTag());

answered Jun 17 '15 at 13:39

Michal Čizmazia

875
1
8
14

score 2 · Answer 3 · answered Mar 04 '15 at 18:32

2

Try this example:

JavaRDD<Tuple2<Integer, String>> mutate = mutateFunction(rdd_world); //goes to a method that generates the RDD with a Tuple2 from a rdd_world RDD
JavaPairRDD<Integer,  String> pairs = JavaPairRDD.fromJavaRDD(mutate);

answered Mar 04 '15 at 18:32

3xCh1_23

1,491
1
20
39

score 2 · Answer 4 · edited Aug 21 '17 at 11:51

Try this to transform JavaRDD into JavaPairRDD. For me It is working perfectly.

JavaRDD<Sensor> sensorRdd = lines.map(new SensorData()).cache();
// transform data into javaPairRdd
JavaPairRDD<Integer, Sensor> deviceRdd = sensorRdd.mapToPair(new PairFunction<Sensor, Integer, Sensor>() {   
    public Tuple2<Integer, Sensor> call(Sensor sensor) throws Exception {
        Tuple2<Integer, Sensor>  tuple = new Tuple2<Integer, Sensor>(Integer.parseInt(sensor.getsId().trim()), sensor);
        return tuple;
    }
});

score 1 · Answer 5 · answered Mar 19 '15 at 12:42

1

Alternatively you can call mapToPair(..) on your instance of org.apache.spark.api.java.JavaRDD.

answered Mar 19 '15 at 12:42

preeze

1,061
1
12
18

Spark: JavaRDD to JavaPairRDD<>

5 Answers5

Linked