Cartesian product of two DStream in Spark

Question

How I can product two DStream in apache streaming like cartesian(RDD<U>) which when called on datasets of types T and U, returns a dataset of (T, U) pairs (all pairs of elements).

One solution is using join as follow that doesn't seem good.

    JavaPairDStream<Integer, String> xx = DStream_A.mapToPair(s -> {
        return new Tuple2<>(1, s);
    });

    JavaPairDStream<Integer, String> yy = DStream_B.mapToPair(e -> {
        return new Tuple2<>(1, e);
    });

    DStream_A_product_B = xx.join(yy);

Is there any better solution? or how i can use Cartesian method of RDD?

isnt it this way http://stackoverflow.com/questions/29034825/cartesian-of-dstream? — Ram Ghadiyaram, Jun 11 '16 at 17:14
@RamPrasadG No! it is cartesian of a DStream with an RDD. But I have two DStream. — Majid Hajibaba, Jun 15 '16 at 06:12

score 0 · Answer 1 · answered Jun 20 '16 at 05:25

I found the answer:

JavaPairDStream<String, String> cartes = DStream_A.transformWithToPair(DStream_B, 
     new Function3<JavaPairRDD<String, String>, JavaRDD<String>, Time, JavaPairRDD<String, String>>() {
        @Override
        public JavaPairRDD<String, String> call(JavaRDD<String> rddA, JavaRDD<String> rddB, Time v3) throws Exception {
            JavaPairRDD<String, String> res = rddA.cartesian(rddB);
            return res;
        }
    });

Cartesian product of two DStream in Spark

1 Answers1

Linked