0

I'm working with Spark's Java API. Now I need to "translate" my Java Application into Scala code because I want to try out Apache Zeppelin. I have a PairDStream (1.1.1.123,(1,1,1,1)) and need to map its pairs to a PairDStream of form (1.1.1,(1,1,1,1)) (cut the "123" in the key of the first pair). Can someone give a clue how to do this or how to map a pair to another one in scala generally? I used the map function to create the first pair. Thank you!

D. Müller
  • 3,336
  • 4
  • 36
  • 84
  • It's not clear what you are trying to do. Are you just trying to cut out the 123, or is that already created and you want to join them together. Can you post a sample and also what you've tried? – Justin Pihony Aug 28 '15 at 14:56
  • I already created a PairRDD with key 1.1.1.123 and value 1,1,1,1. Now i want to map this pairs to new ones with the shorter key 1.1.1. I want to remove the fourth digit of the key, no joining. Thanks – D. Müller Aug 28 '15 at 15:32
  • Did you even try using another map...? It sounds pretty straight forward then.... – Justin Pihony Aug 28 '15 at 17:09
  • I tried another map function call. But the problem I had was to get the key and the value object of the pair, so that I was able to map them to a new pair later on... The answer below was the information I needed, thanks for your clues! – D. Müller Aug 31 '15 at 07:58

1 Answers1

0

Try this (assuming your IP address is a String):

val newPairs = oldPairs.map { case (ipAddress, values) => (ipAddress.substring(0, ipAddress.lastIndexOf(".")) , values) }
Jeff L
  • 141
  • 1
  • 1
  • 7
  • The map {case (k,v) => (..., ...) } did it for me, thank you! I didn't know about the case-statement to get the key / value part of the pair. – D. Müller Aug 31 '15 at 07:59