How to read redis map in spark using spark-redis

Question

I have a normal scala map in Redis (key and value). Now I want to read that map in one of my spark-streaming program and use this as a broadcast variable so that my slaves can use that map to resolve key mapping. I am using spark-redis 2.3.1 library, but now sure how to read that.

Map in redis table "employee" -

name   |    value
------------------
123         David
124         John
125         Alex

This is how I am trying to read in spark (Not sure if this is correct- please correct me) --

 val loadedDf = spark.read
  .format("org.apache.spark.sql.redis")
  .schema(
    StructType(Array(
      StructField("name", IntegerType),
      StructField("value", StringType)
    )
  ))
  .option("table", "employee")
  .option("key.column", "name")
  .load()
loadedDf.show()

The above code does not show anything, I get empty output.

Could you please check my below answer? – Moustafa Mahmoud May 11 '19 at 13:34 — Moustafa Mahmoud, May 11 '19 at 13:34
Did the below answer worked with you? – Moustafa Mahmoud May 13 '19 at 18:49 — Moustafa Mahmoud, May 13 '19 at 18:49

score 2 · Answer 1 · answered May 11 '19 at 13:32

You could use the below code to your task but you need to utilize Spark Dataset (case Dataframe to case class) to do this task. Below is a full example to read and write in Redis.

object DataFrameExample {

  case class employee(name: String, value: Int)

  def main(args: Array[String]): Unit = {
    val spark = SparkSession
          .builder()
          .appName("redis-df")
          .master("local[*]")
          .config("spark.redis.host", "localhost")
          .config("spark.redis.port", "6379")
          .getOrCreate()

    val personSeq = Seq(employee("John", 30), employee("Peter", 45)
    val df = spark.createDataFrame(personSeq)

    df.write
      .format("org.apache.spark.sql.redis")
      .option("table", "person")
      .mode(SaveMode.Overwrite)
      .save()

    val loadedDf = spark.read
                        .format("org.apache.spark.sql.redis")
                        .option("table", "person")
                        .load()
    loadedDf.printSchema()
    loadedDf.show()
  }
}

Output is below

root
 |-- name: string (nullable = true)
 |-- value: integer (nullable = false)

+-----+-----+
| name|value|
+-----+-----+
| John| 30  |
|Peter| 45  |
+-----+-----+

You could also check more details in Redis documentation

thanks for the answer, but what if in case I want to read using readStream — thedevd, Jun 04 '19 at 10:31

How to read redis map in spark using spark-redis

1 Answers1