3

In Spark you can cast a LongType without issues to TimestampType. And after that there are no issues casting a TimestampType to a DateType.

The flow 

long -> timestamp -> date

therefor is possible. However, it is not possible to cast a LongType directly to DateType. So

long -> date

is not allowed.

Internally a TimestampType is just a long and a DateType is just an int.

Thus...

long -> long -> int // this is fine
long -> int // this is not fine

Why is that? Why doesn't spark allow casting to a date directly from a long?

Tim
  • 2,000
  • 4
  • 27
  • 45
  • 2
    Ambiguity for starters. Furthermore internal representation doesn't reflect external one (timestamp has millisecond precision, while long during casting is interpreted as seconds). – zero323 Mar 15 '17 at 14:55
  • So why does casting from long to timestamp cause no problems? Sure, timestamp has millisecond precision, date has day precision. But a long is just a long. So whatever the long is casted to, there must be some kind of explicit conversion... why isn't there one for `long -> date` since it's inherently still a loss of information, regardless if `timestamp` is in between? – Tim Mar 16 '17 at 07:04
  • If you're looking for a strict reason why it cannot be done I don't think there is one. Effectively we have well established conventions of representing timestamps in (milli)seconds but not such thing for days. Ambiguity is a problem as well - should we interpret integer as (milli)seconds or days? You can also check http://stackoverflow.com/q/42628287/1560062 to see that internal representation is not that relevant since internally it won't be used directly anyway. – zero323 Mar 16 '17 at 11:06

1 Answers1

-2
val DEFAULT_DATE_FORMAT: String = "yyyy-MM-dd"
val unixDate: Date = new Date(if (ts > Int.MaxValue) ts else ts * 1000)
val dateFormatter: SimpleDateFormat = new SimpleDateFormat(DEFAULT_DATE_FORMAT)
val date: String = dateFormatter.format(unixDate)
noamt
  • 7,397
  • 2
  • 37
  • 59
Ting Jia
  • 197
  • 1
  • 1
  • 6