How do I connect to and write a csv file to a remote instance of Databricks Apache Spark from Java?

Question

I'm trying to connect to a remote instance of Databricks and write a csv file to a specific folder of the DBFS. I can find bits and pieces here and there but I'm not seeing how to get this done. How do I add the file to DBFS on a remote Databricks instance from a Java program running on my local machine?

I'm currently using a community instance I created from here: https://databricks.com/try-databricks

This is the url for my instance (I'm guessing the "o=7823909094774610" is identifying my instance).
https://community.cloud.databricks.com/?o=7823909094774610

Here's some of the resources I'm looking at trying to resolve this but I'm still not able to get off of the ground:

The Databricks Connect documentation: This talks about connecting but not specifically from Java. It gives and example of "connecting Eclipse" to data bricks that seems to be how to get the jar dependency for this (side question, is there a mvn version of this?). https://docs.databricks.com/dev-tools/databricks-connect.html#run-examples-from-your-ide
Some Java sample code: Doesn't seem to have an example of connecting to a remote Databricks instance https://www.programcreek.com/java-api-examples/index.php?api=org.apache.spark.sql.SparkSession
Databricks File System (DBFS) Documentation: Gives a good overview of file functions but doesn't seem to talk specifically about how to connect from a remote Java application and write the file to the Databricks instance from the Java application https://docs.databricks.com/data/databricks-file-system.html
FileStore documentation: Gives a good overview of file store but again doesn't seem to talk specifically about how to do this from a remote Java application https://docs.databricks.com/data/filestore.html

databricks connect isn't supported for community edition – Alex Ott Aug 21 '20 at 12:43 — Alex Ott, Aug 21 '20 at 12:43

score 2 · Accepted Answer · answered Aug 20 '20 at 21:09

You could take a look at the DBFS REST API, and consider using that in your Java application.

If a Java solution is not required, then you could also take a look at the databricks-cli. After installing it with pip (pip install databricks-cli) you simply have to:

Configure the CLI by running: databricks configure
- Host: https://community.cloud.databricks.com/?o=7823909094774610
- Username: <your username>
- Password: <your password>
Copy the file to DBFS by running: databricks fs cp <source> dbfs:/<target>

Thanks Bram, this is a great answer! I'm looking into the REST API now. I'll let you know how it works out. — John, Aug 21 '20 at 13:32

How do I connect to and write a csv file to a remote instance of Databricks Apache Spark from Java?

1 Answers1