2

I am trying to write pyspark dataframe to Azure Postgres Citus (Hyperscale). I am using latest Postgres JDBC Driver and I tried writing on Databricks Runtime 7,6,5.

df.write.format("jdbc").option("url","jdbc:postgresql://<HOST>:5432/citus?user=citus&password=<PWD>&sslmode=require" ).option("dbTable", table_name).mode(method).save()

This is what I get after running the above command org.postgresql.util.PSQLException: SSL error: Received fatal alert: handshake_failure

I have already tried different parameters in the URL and unders the option as well, but no luck so far. However, I am able to connect to this instance using my local machine and on databricks driver/notebook using psycopg2 Both the Azure Postgres Citus and Databricks are in the same region and Azure Postgres Citus is public.

anshul_cached
  • 684
  • 5
  • 18

1 Answers1

7

It worked by overwriting the java security properties for driver and executor

spark.driver.extraJavaOptions -Djava.security.properties= spark.executor.extraJavaOptions -Djava.security.properties=

Explanation:

What is happening in reality is that the “security” variable of the JVM is reading by default the following file (/databricks/spark/dbconf/java/extra.security) and in this file there are some TLS algorithms that are being disabled by default. That means that if I edit this file and replace the TLS cyphers that work for PostGres citus for an empty string that should also work.

When I set this variable to the executors (spark.executor.extraJavaOptions) it will not change the default variables from the JVM. The same does not happen for the driver which overwrites and so it starts to work.

Note: We need to edit this file before the variable is read and so the init script is the only way of accomplishing that.

anshul_cached
  • 684
  • 5
  • 18
  • 1
    Just for reference, the default disabledAlgorithms in Databricks 7.3 are: jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, EC keySize < 224, 3DES_EDE_CBC, anon, NULL, GCM – – Carlos David Peña Feb 12 '21 at 14:59
  • Thanks a lot! You solved my issue (connection from Azure Databricks 9.1 LTS to PostgreSQL Flexi server with TLS 1.2) – Mihails Akimenko Jan 13 '22 at 14:51