0

I am completely new to Flink. May this question is repeated but found only one link and that is not understandable for me.

https://stackoverflow.com/a/44294980/6904987

I stored Data in Redis in Key Value format example Key is UserId and UserInfo is value. Written below code for it.

class RedisExampleMapper extends RedisMapper[(String, String)] {
  override def getCommandDescription: RedisCommandDescription = {
    new RedisCommandDescription(RedisCommand.HSET, "HASH_NAME")
  }

  override def getKeyFromData(data: (String, String)): String = data._1

  override def getValueFromData(data: (String, String)): String = data._2
}


val env = StreamExecutionEnvironment.getExecutionEnvironment
    val conf = new FlinkJedisPoolConfig.Builder().setHost("IP").build()
    val streamSink = env.readTextFile("/path/useInformation.txt").map(x => {
          val userInformation = x.split(",")
          val UserId = userInformation(0)
          val UserInfo  = userInformation(1)
          (UserId , UserInfo)
        })
val redisSink = new RedisSink[(String, String)](conf, new RedisExampleMapper)
streamSink.addSink(redisSink)

Sample Data:

12 "UserInfo12"

13 "UserInfo13"

14 "UserInfo14"

15 "UserInfo15"

I want to feteched data from redis using Flink based on key . example 14 should return "UserInfo14". Output should print in Flink Log file or terminal whatever it is.

Thanks in advance.

Darshan Manek
  • 155
  • 1
  • 2
  • 12
  • Please extend the description of what you want to do. How much data will you read from Redis? Your question suggests that this might just be a single (or very few) records. If that is true, Flink is not the right tool for the job as it adds too much overhead. Flink is a framework for processing large amounts of data (streaming or batch) in parallel. The framework adds too much overhead for just fetching a single value. – Fabian Hueske Apr 16 '18 at 08:50

2 Answers2

0

Extending on the answer in https://stackoverflow.com/a/44294980/6904987.

Add the source with env.addSource(new RedisSource(data structure name)).

You have to implement yourself the RedisSource that connects to a Redis database, reading the records from a Redis data structure.

The implementation depends. Either you consume from Redis through polling or you subscribe to Redis, emitting events from the source whenever you get them from Redis.

You can check the general SourceFunction example and documentation available here: https://ci.apache.org/projects/flink/flink-docs-release-1.5/api/java/org/apache/flink/streaming/api/functions/source/SourceFunction.html

Alex
  • 839
  • 1
  • 12
  • 24
0

If you want to query Redis for key-value search, you can use a Redis client inside your transformations. For example, Jedis can be used to query Redis if you are using Java with Flink.

avidlearner
  • 243
  • 3
  • 16