Migrate (on-prem) SQL data to Azure with Databricks (JDBC)

Question

Is it possible to use the JDBC connector https://docs.databricks.com/data/data-sources/sql-databases.html in order to get data from local SQL server. (and export it to delta lake)

Using:

jdbcUrl = "jdbc:mysql://{0}:{1}/{2}".format(jdbcHostname, jdbcPort, jdbcDatabase)
connectionProperties = {
  "user" : jdbcUsername,
  "password" : jdbcPassword,
  "driver" : "com.mysql.jdbc.Driver"
}

SQL Server or MySQL? Your question asks about SQL Server but you're using a MySql jdbc url. — David Browne - Microsoft, Sep 18 '20 at 13:29

score 1 · Answer 1 · answered Sep 18 '20 at 14:12

Irrespective if you have MySql or SQL Server, Databricks driver supports both as outlined in the article you linked. From the perspective of access to on-prem - the answer is yes, however Databricks must be able to connect to it. Usually this will mean deploying your Databricks clusters into your VNET which has access to your on-prem resources, e.g. following the guidance here

Alternatively you could use Azure Data Factory self-hosted integration runtime to move the data to a staging/"Bronze" storage in the cloud and pick it up with a Databricks task to move it to a Delta table.

I double on moving data instead of using JDBC – Alex Ott Sep 18 '20 at 15:53 — Alex Ott, Sep 18 '20 at 15:53

Migrate (on-prem) SQL data to Azure with Databricks (JDBC)

1 Answers1