Convert an RDD into a key value pair RDD, with the values being in a List

Question

I have an pairRDD with tuples being in the following form:

[(1,"b1","c1","d1","e1"), (2,"b2","c2","d2","e2"), ...

What I want is to transform the above into a key-value pair RDD, where the first field will be the key, and the second field a list of strings (value). i.e. I want to turn it to the form:

[(1,["b1","c1","d1","e1"]), (2,["b2","c2","d2","e2"]), ...

After this, is it then possible to access any field that I want?

For example, can I access the tuple (1,["b1","c1","d1","e1"]), then extract only the field d1?

This looks like a `HashMap>` or `LinkedHashMap>` if you want to preserver the keys' order. — Titus, Jul 10 '16 at 19:02
Oh, I've misunderstood, I thought you wanted to convert a RDD object to a normal Java object, as a RDD object, you can use `JavaPairRDD>`. — Titus, Jul 10 '16 at 19:11

YoYo · Accepted Answer · 2016-07-10T21:34:05.847

4

If you have an RDD with Tuples, however the Tuples are represented, you can use mapToPair to transform your RDD of Tuple into a PairRDD with Key and Value as preferred.

In Java 8 this could be

JavaPairRDD<Integer,List<String>> r = 
  rddOfTuples.mapToPair((t)->new Tuple2(
      extractKey(t),
      extractTuples(t)
  ));

Note that this operation will introduce a shuffle.

To state the obvious, extractKey and extractTuples are to be methods to be implemented extracting the parts of the original tuple as needed.

With my limited knowledge of Scala Tuples, And assuming the input is something like scala.Tuple5<String,Integer,Integer,Integer,Integer>, this could be:

JavaPairRDD<Integer,List<String>> r = 
  rddOfTuples.mapToPair((t)->new Tuple2(
      t._1,
      Arrays.asList(t._2,t._3,t._4,t._6)
  ));

If however, you do not know beforehand the arity (number of elements) of your Tuple, then in scala terms, it is a Product. To access your elements dynamically, you will need to use the Product interface, with a choice of:

int productArity()
Object productElement(int n)
Iterator<Object> productIterator()

Then it becomes a regular Java exercise:

JavaPairRDD<Integer,List<String>> r = 
  rddOfTuples.mapToPair((t)->{
    List<String> l = new ArrayList<>(t.productArity()-1);
    for (int i = 1; i < t.productArity(); i++) {
      l.set(i-1,t.productElement(i));
    }
    return new Tuple2<>(t._1,l);
  }));

I hope I have it all right ... this code above is untested/uncompiled ... So if you can get it to work with corrections, then feel free to apply the corrections in this answer ...

edited Jul 10 '16 at 21:34

answered Jul 10 '16 at 20:21

YoYo

9,157
8
57
74

but the problem here is that i want the value as a list – hammad Jul 10 '16 at 20:27
means the tuple ll be defined as (1,["hh","gg","hh"]) and 1 here represents the key – hammad Jul 10 '16 at 20:30
wold you modify your code please to show me how to do it? – hammad Jul 10 '16 at 20:31
no this is just an example it can be any type bcz i use diffrent data somtimes it s scala.Tuple5 or scala.Tuple3 or... so i need a standard function – hammad Jul 10 '16 at 21:01
What we really need is a better consolidation between core java and scala features ... I wish that Java inherits some of the best of scala (native tuple and list support) and Python (slicing) with none of the complexity, but consistent with java syntax. – YoYo Jul 10 '16 at 21:39
wait a min .. your input tuple is actually a String? So you start with a RDD ? – YoYo Jul 10 '16 at 22:50
no it was a fault, i start with a JavaPairRDD – hammad Jul 10 '16 at 22:55
but i can start with a RDD? is it easier? – hammad Jul 10 '16 at 22:56
Much easier, handling the tuple is really cumbersome. – YoYo Jul 10 '16 at 23:02
please, would you tell me how to do it , and i m sorry – hammad Jul 10 '16 at 23:11

score 1 · Answer 2 · answered Jul 10 '16 at 20:32

1

You could try using a map function, eg in Scala:

rdd.map { case (k,v1,v2,v3,v4) => (k,(v1,v2,v3,v4)) }

Or rdd.groupBy could also be used but this could be inefficient on large data sets.

answered Jul 10 '16 at 20:32

PJ Fanning

953
5
13

how can i do it in java – hammad Jul 10 '16 at 20:35
I'm not as familiar with using Spark in Java but this code might help: http://www.programcreek.com/java-api-examples/index.php?source_dir=oryx-master/app/oryx-app-mllib/src/main/java/com/cloudera/oryx/app/batch/mllib/als/Evaluation.java - you can create a Function that takes Tuple5 as input and returns a Tuple2> – PJ Fanning Jul 10 '16 at 20:43
Note that mapping it that way, still keeps it a as a non-keyed RDD, it will just be a new RDD of tuples. – YoYo Jul 10 '16 at 21:30
but what if v1 .. v7? that's the more important issue – thebluephantom Feb 12 '18 at 20:41

Convert an RDD into a key value pair RDD, with the values being in a List

2 Answers2