Using Composite Key in DSE4.8 spark streaming

Question

can anybody share example in java with composite primary key in spark sql to save data in Cassandra. With CQL I can do it easily. But what i am doing is inside a spark job which stream time series data from Kafka stream and saving it to cassandra as rawdata. Then using dataframe i can join this table with related tables and perform refining of data using business rules and then store data in refined data table.

Answer 1

I was able to solve the first part of my question . That is, saving data to Cassandra table with composite primary key using spark-sql. Basically if we already created a table with composite primary key, it can be done easily. There is no difference whether the key is composite or not. I created a table with primary key imei and date with time using CQL in cassandra . Then following code worked perfectly for me.

rowData.foreachRDD(rdd -> {
             if(rdd.count()>0){
             SQLContext sqlContext = SQLContext.getOrCreate(sc);
             Map<String, String> options = new HashMap<String, String>();
             options.put("table","data");
             options.put("keyspace","newavlview");

                org.apache.spark.sql.DataFrame   wordsDataFrame = sqlContext.createDataFrame(rdd, XMLRowBean.class);

                wordsDataFrame.
                write().format("org.apache.spark.sql.cassandra")
                .options(options).mode(SaveMode.Append)
                .save();


             }