Window functions / scala / spark 1.6

Question

I would like to use a window function in Scala.

I have a CSV file which is the following one :

id;date;value1
1;63111600000;100
1;63111700000;200
1;63154800000;300

When I try to apply a window function over this data frame, sometimes it works and sometimes it fails:

val df = loadCSVFile() 
val tw = Window.orderBy(date).partitionBy(id).rangeBetween(-5356800000,0) 
df.withColumn(value1___min_2_month, min(df.col("value1")).over(tw))
+---+-----------+--------------------+
| id|       date|value1___min_2_month|
+---+-----------+--------------------+
|  1|63111600000|                 100|
|  1|63111700000|                 100|
|  1|63154800000|                 100|
+---+-----------+--------------------+

So it works ! But when I try with a bigger number (which contains row of the previous exemple) I have the following result

val tw = 

Window.orderBy(date).partitionBy(id).rangeBetween(-8035200000,0) \n
df.withColumn(value1___min_3_month, min(df.col("value1")).over(tw)) 
+---+-----------+--------------------+
| id|       date|value1___min_3_month|
+---+-----------+--------------------+
|  1|63111600000|                null|
|  1|63111700000|                null|
|  1|63154800000|                null|
+---+-----------+--------------------+

score 2 · Answer 1 · answered Feb 02 '17 at 16:47

2

Suffix your number with L:

scala> -10000000000
<console>:1: error: integer number too large
-10000000000
 ^

scala> -10000000000L
res0: Long = -10000000000

answered Feb 02 '17 at 16:47

pedrofurla

12,763
1
38
49

My problem is that this number is in fact in a variable (long type) and with a variable it doesn't work – Gaëlle Hisler Feb 03 '17 at 09:14
Then you just pass the variable. – pedrofurla Feb 03 '17 at 15:16

Window functions / scala / spark 1.6

1 Answers1