How to send the data inside a list to kafka as a record

Question

I'm new to scala, I have a List like this

newlist = List([Date,Open,High,Low,Close,Volume,Name], [2012-08-13T00:00:00.000Z,92.29,92.59,91.74,92.4,2075391.0,MMM], [2012-08-14T00:00:00.000Z,92.36,92.5,92.01,92.3,1843476.0,MMM], [2012-08-15T00:00:00.000Z,92.0,92.74,91.94,92.54,1983395.0,MMM], [2012-08-16T00:00:00.000Z,92.75,93.87,92.21,93.74,3395145.0,MMM], [2012-08-17T00:00:00.000Z,93.93,94.3,93.59,94.24,3069513.0,MMM], [2012-08-20T00:00:00.000Z,94.0,94.17,93.55,93.89,1640008.0,MMM])

I use the below code to send the data inside a list to Kafka

      val today = Calendar.getInstance.getTime
      val formatter = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss")
      val key = UUID.randomUUID().toString().split("-")(0)

      val value = formatter.format(today) + "," + newList

      val data = new ProducerRecord[String, String](topic, key, value)

      println(data.value())
      producer.send(data)

and my output is like below:

2020-05-12 14:56:41,List([Date,Open,High,Low,Close,Volume,Name], [2012-08-13T00:00:00.000Z,92.29,92.59,91.74,92.4,2075391.0,MMM], [2012-08-14T00:00:00.000Z,92.36,92.5,92.01,92.3,1843476.0,MMM], [2012-08-15T00:00:00.000Z,92.0,92.74,91.94,92.54,1983395.0,MMM], [2012-08-16T00:00:00.000Z,92.75,93.87,92.21,93.74,3395145.0,MMM], [2012-08-17T00:00:00.000Z,93.93,94.3,93.59,94.24,3069513.0,MMM], [2012-08-20T00:00:00.000Z,94.0,94.17,93.55,93.89,1640008.0,MMM])
2020-05-12 14:56:42,List([Date,Open,High,Low,Close,Volume,Name], [2012-08-13T00:00:00.000Z,92.29,92.59,91.74,92.4,2075391.0,MMM], [2012-08-14T00:00:00.000Z,92.36,92.5,92.01,92.3,1843476.0,MMM], [2012-08-15T00:00:00.000Z,92.0,92.74,91.94,92.54,1983395.0,MMM], [2012-08-16T00:00:00.000Z,92.75,93.87,92.21,93.74,3395145.0,MMM], [2012-08-17T00:00:00.000Z,93.93,94.3,93.59,94.24,3069513.0,MMM], [2012-08-20T00:00:00.000Z,94.0,94.17,93.55,93.89,1640008.0,MMM])

But I want my output to be like this(each value of a list as row):

2020-05-12 14:56:41,Date,Open,High,Low,Close,Volume,Name
2020-05-12 14:56:42,2012-08-13T00:00:00.000Z,92.29,92.59,91.74,92.4,2075391.0,MMM
2020-05-12 14:56:43,2012-08-14T00:00:00.000Z,92.36,92.5,92.01,92.3,1843476.0,MMM

How can we do that? Please ignore my mistakes

Does this answer your question: https://stackoverflow.com/questions/58420634/how-to-efficiently-send-messages-out-of-a-seqstring-to-kafka — Michael Heil, May 12 '20 at 12:04
I don't really understand your types. This is a List of what? — mfirry, May 12 '20 at 12:05

score 1 · Accepted Answer · answered May 12 '20 at 17:05

1

ProducerRecord is a instance for a single data If You want to send list of data, multiple call 'send' method

newlist.map(row => 
    producer.send(new ProducerRecord[String, String](topic, key, row.mkString))

answered May 12 '20 at 17:05

bistros

1,139
1
9
23

Thanks for the help but it gives me output without space like this: DateOpenHighLowCloseVolumeName 2012-08-13T00:00:00.000Z92.2992.5991.7492.42075391.0MMM, I want space or comma between each values like this: Date Open High Low Close Volume Name 2012-08-13T00:00:00.000Z 92.29 92.59 91.74 92.420753 91.0MMM Can you help me with that – Fashil May 13 '20 at 11:47
that is sacla programming grammer issue. If You use `List("A","B").mkString(", ")` then You see "A, B" – bistros May 13 '20 at 15:16

How to send the data inside a list to kafka as a record

1 Answers1