How can I connect a .net application to read a stream from a Delta table in Azure Databricks?

Question

How can I write e.g. a console application in .Net that would read a delta table or open a stream to a delta table in Azure Databricks.

I've tried this code

var spark = SparkSession
    .Builder()
    .AppName("Streaming example with a UDF")
    .Config("spark.databricks.service.address", "https://adb-somenumericid.azuredatabricks.net/")
    .Config("spark.databricks.service.token", "mypat")
    .Config("spark.databricks.service.clusterId", "0118-xxxxx-yyyyyy")
    .Config("spark.databricks.delta.preview.enabled", true)
    .GetOrCreate();

var table = spark.Read().Format("delta").Table("mycatalog.mydb.mytable");
table.Show();

but I get the error

[MSF002237] [Exception] [JvmBridge] No connection could be made because the target machine actively refused it. 127.0.0.1:5567

How can I get the code to connect to my Databricks cluster instead of localhost?

score 0 · Answer 1 · answered Mar 15 '23 at 13:22

I tried to reproduce the issue with below code:

using Microsoft.Spark.Sql;

SparkSession spark = SparkSession.Builder()
    .AppName("Read Delta table in .NET")
    .Config("spark.databricks.service.address", "https://<databricksID>.azuredatabricks.net")
    .Config("spark.databricks.service.token", "<AccessToken>")
    .GetOrCreate();

   
DataFrame deltaTable = spark.Read()
    .Format("delta")
    .Table("hive.warehouse.empp");


deltaTable.Show();

I got the same error:

enter image description here

I followed below procedure to resolve the issue.

I installed .net core SDK and set the path of .net core sdk in environment variables and install java and set path for java and downloaded apache spark and extract the files into C:\bin\spark and set below environment variables HADOOP_HOME with C:\bin\spark, SPARK_HOME with C:\bin\spark Path with C:\bin\spark . Installed Microsoft worker and installed to C:\bin\runner set that that path as environment variable. I checked for confirmation of spark by using

spark-submit --version

I got below output:

enter image description here

I downloaded and installed the winutils to the spark path. I created run the console applicaton by using below format in the console app path in command prompt

spark-submit.cmd --class org.apache.spark.deploy.dotnet.DotnetRunner --master local <jarpath>\mrosoft-spark-2-4_2.11-<version>.jar dotnet sparkconsole.dll

It executed without any error. For more information you can check this and you can follow this.

How can I connect a .net application to read a stream from a Delta table in Azure Databricks?

1 Answers1