Stack Overflow
Stack Overflow

Questions tagged [spark-redis]

26 questions
1
vote
1 answer

How to turn redis into spark dataset or dataframe?

I am trying to use redis as the source of spark sql, but got stuck with how to transform the rdd. Below are my codes: RDD> rdd1 = rc.fromRedisKV("user:*",3,redisConfig); JavaRDD userRDD = rdd1.toJavaRDD().map(new…
Simpson Yang
  • 81
  • 5
0
votes
0 answers

How to setup spark-redis in python?

I'm building an stream-processing app and I thought redis could be a good addition. I'm trying to read the data files into dataframes and then loading them to redis. I've defined the SparkSession as follows: spark_session = SparkSession \ …
Francisco Chacón Rubio
  • 11
  • 1
0
votes
1 answer

Redis return null value to id column

I'm write a dataframe to redis and read the data with spring boot application. The information retry successfull, but the id column return null like that: { "nu_document": null, "qt_negative": "1000000000" }
Alan Miranda
  • 125
  • 1
  • 8
0
votes
0 answers

What is the latest Spark-Redis connector version we can install in databricks cluster?

What is the latest spark-redis version we can install in databricks cluster ? I will be using python/pyspark to write the dataframe to redis. My Databricks runtime :10.2 (includes Apache Spark 3.2.0, Scala 2.12) In Databricks, we can see only…
Sai
  • 31
  • 2
0
votes
1 answer

Save kafka stream dataframe to Redis in Databricks after data transformation

I am using pyspark to direct the kafka streams to redis after performing aggregations on the data. The final output is a streaming datafame. The code I connect to kafka streams. (You might find my code is a layman job, Please ignore) app_schema =…
Vamsi Nimmala
  • 497
  • 1
  • 7
  • 19
0
votes
0 answers

Java - How to convert Iterable to JavaPairRDD

I have a JavaPairRDD of (String, Iterable[(String, String)]) and I want this to be converted into an JavaPairRDD of (String, RDD[String, String]), so that i can use the reduceByKey function to the internal JavaPairRDD. In this example, I have a…
slash
  • 9
  • 1
0
votes
1 answer

How to read redis map in spark using spark-redis

I have a normal scala map in Redis (key and value). Now I want to read that map in one of my spark-streaming program and use this as a broadcast variable so that my slaves can use that map to resolve key mapping. I am using spark-redis 2.3.1…
thedevd
  • 683
  • 11
  • 26
0
votes
1 answer

How to use redis in Spark Streaming

I am building an application that reads json elements from a list in redis aand streams them using spark. Here is what i have writen: public void readTheStream() throws UnknownHostException, IOException { SparkConf sparkConf = new…
thecoder
  • 29
  • 5
0
votes
0 answers

Spark streaming Redis Read Time Out with Scala

While i'm reading table from redis getting this below error. Below code normally working well. val readDF= spark.sparkContext.fromRedisKeyPattern(tableName,5).getHash().toDS() Normally it's working for less than 2 million rows. But if i'm reading…
Beyhan Gul
  • 1,191
  • 1
  • 15
  • 25
0
votes
0 answers

Storm + Redis or Storm Trident or Spark Streaming

i am going to build stream processing systems. Using Kafka for message transmission. Stream processing can be done with storm trident, storm or spark streaming. But couldn't find the best answer. Stream Processing will be simple if conditions at…
Joker
  • 96
  • 6
-1
votes
1 answer

Retrieve a String type column from a Spark Dataset as String variable, to pass that as a 'key' for the Redis cache

I am trying to use spark streaming to read the data from a kafka topic. The message from kafka is a JSON which i am storing below in the value column of the dataset as String. Sample message : Just a sample, actual json is complex { "Name":…
Bauddhik
  • 9
  • 4
1
2