0

I'm running code in Databricks but somehow unixtimestamp values are not different. On debugging I get to know, because databricks is configured with UTC timezone. Please advise how can I change default timezone to EST in Databricks.

Vinitkumar
  • 111
  • 5
  • 18

2 Answers2

1

You can set it in the cluster -> configuration -> Advanced Option -> spark, set the spark parameter: spark.sql.session.timeZone Hongkong. It will impact any sql statement time zone.

If you want to set the time zone all data nodes, u need to add an initial script and set the unix time zone. so that some thing like the python datetime.dateime.now(), would be the set timezone.

dsk
  • 1,863
  • 2
  • 10
  • 13
  • Thanks I worked. Can you specify sample command which I can use to set default timezone in each node to EST. – Vinitkumar Jul 12 '21 at 14:12
0

you can also set time zone while defining spark session :

from pyspark.sql import SparkSession as PySparkSession
from pyspark import SparkConf


TIMEZONE = 'AFRICA/CAT'

class SparkSession:

    def __init__(self, config=[]):
        config.append(tuple(('spark.sql.session.timeZone',TIMEZONE)))
        self.conf = SparkConf().setAll(config)
        self.spark = (PySparkSession.builder
                      .config(conf=self.conf)
                      .enableHiveSupport()
                      .getOrCreate())
Anujsharma
  • 47
  • 8