0

Trying to reduceByKey on multidimensional list using scala , such that tuples are appended to parent tuple and thus generate multidimensional tuple.

In python I am trying to append to multidimensional list as follows and it works perfectly :

.map(lambda z:(z[1][0][1],[[z[1][0][1],str(z[1][0][2]),str(z[1][0][3]),z[1][0][0].strftime('%Y-%m-%dT%H:%M:%SZ'),z[1][1]]])).reduceByKey(lambda a,b:a+b)

But in scala I am unable to use reduceByKey , I am trying following :

.map(t => (t._2._1._2,((t._2._1._2,t._2._1._3,t._2._1._4,t._2._1._1,t._2._2)))).reduceByKey(t,y => t++y)

Any hints in right direction are also welcome!

Harshit
  • 1,207
  • 1
  • 20
  • 40

1 Answers1

2

Scala Tuple*, unlike Python tuple, is not a collection. It is a Product. Technically it represents n-fold Cartesian product of, possible heterogeneous, sets of values. Scala Tuples cannot be concatenated and cannot contain more than 22 elements.

If you want to collect values per key you should either use some type of collection or even bettergroupByKey.

See also: How should I think about Scala's Product classes?

Community
  • 1
  • 1
zero323
  • 322,348
  • 103
  • 959
  • 935