How to parallelize a list of key value pairs to JavaPairRDD in Apache spark Java API?

Question

I have List of Key,value pairs such as List((A,1),(B,2),(C,3)) in heap memory. How can I parallelize this list to create a JavaPairRDD? In scala : val pairs = sc.parallelize(List((A,1),(B,2),(C,3)). Likewise, Is there any way with java API?

I referred the manual. I know how to do it with scala and python. is there any way doing it with java? — Sandeep Veerlapati, Apr 29 '16 at 09:45

score 2 · Accepted Answer · edited May 23 '17 at 11:48

2

I found the answer. First store the List of tuples in JavaRDD and then convert it to JavaPairRDD.

    List<Tuple2> data =  Arrays.asList(new Tuple2("panda", 0),new Tuple2("panda", 1));
    JavaRDD rdd = sc.parallelize(data);
    JavaPairRDD pairRdd = JavaPairRDD.fromJavaRDD(rdd);

Have a look at this answer

edited May 23 '17 at 11:48

Community

1
1

answered May 02 '16 at 04:38

Sandeep Veerlapati

121
2
11

score 1 · Answer 2 · edited Aug 03 '17 at 02:57

1

I can see this one working for me

sc.parallelizePairs(Arrays.asList(new Tuple2("123","123")));

edited Aug 03 '17 at 02:57

Andy

49,085
60
166
233

answered Aug 02 '17 at 20:35

pranaygoyal02

181
1
10

banjara · Answer 3 · 2016-04-29T17:14:39.407

0

Parallelized collections are created by calling JavaSparkContext’s parallelize method on an existing Collection in your driver program. The elements of the collection are copied to form a distributed dataset that can be operated on in parallel.

List data = ......;
JavaRDD rdd = sc.parallelize(data);

edited Apr 29 '16 at 17:14

answered Apr 29 '16 at 07:10

banjara

3,800
3
38
61

By using the above lines you can only store the elements not the key value pairs. And also I am trying to create JavaPairRDD not JavaRDD – Sandeep Veerlapati Apr 29 '16 at 09:15
1

Although this code may answer the question, providing additional context regarding _why_ and/or _how_ it answers the question would significantly improve its long-term value. Please [edit] your answer to add some explanation. – Toby Speight Apr 29 '16 at 14:54
@SandeepVeerlapati if type of your list is tuple, I think spark will create pairedRDD – banjara Apr 29 '16 at 17:16
@shekar what you are saying is correct in case of scala api. – Sandeep Veerlapati May 02 '16 at 03:59

score 0 · Answer 4 · answered Oct 14 '16 at 16:51

0

Convert Tuple into List with below code snippet.
Tuple2<Sensor, Integer> tuple = new Tuple2<Sensor, Integer>(arg0._2, 1);
                List<Tuple2<Sensor, Integer>> list = new ArrayList<Tuple2<Sensor, Integer>>();
                list.add(tuple);

answered Oct 14 '16 at 16:51

Rajeev Rathor

1,830
25
20

How to parallelize a list of key value pairs to JavaPairRDD in Apache spark Java API?

4 Answers4