0

I am preparing Spark with python program which inserts data from 2 tables based on joins. The last column of the target table has a timestamp field which will have the value of create timestamp.

I tried current_timestamp and from_unixtime(unix_timestamp()). Both the functions does not seem to work. I tried now().

e.g., HiveContext(sc).sql("SELECT " + from_unixtime(unix_timestamp()) + " ") This statement errors in pyspark with "NameError: name 'from_unixtime' is not defined" I have imported the pyspark.sql.function

Is there a way to insert timestamp value to the target table? My query contains insert with select from 2 tables which I am running in HiveContext.

Thank in advance!!!

Aavik
  • 967
  • 19
  • 48
  • I tried the below code and I am getting: TypeError: 'Column' object is not callable import pyspark.sql.functions as func HiveContext(sc).sql("SELECT " + func.from_unixtime(func.unix_timestamp()) + " "). Could someone say what is wrong with this query. Thanks!!! – Aavik Jun 29 '16 at 00:38

1 Answers1

0

Used within double quotes as below and it worked:

HiveContext(sc).sql("SELECT from_unixtime(unix_timestamp())")

Aavik
  • 967
  • 19
  • 48