How does writing data from spark to vertica work?

Question

Are there any good online resources to learn how writing data from Spark to Vertica works? I'm trying to understand why writing to a Vertica database is slow.

This is my basic workflow:

Create a SparkContext. I'm using the class pyspark.sql.SQLContext to create one.
From SQLContext, using the read method to get DataFrameReader interface under 'jdbc' format.

df = self._sqlContext.read.format('jdbc').options(url=self._jdbcURL, dbtable=subquery).load()

Read entries from a Vertica database using jdbc connection (call it dbA)
Write those entries into another Vertica database using the SparkContext in Step 1 (call it dbB)

Right now it's just a simple read from dbA and write to dbB. But writing 50 entries takes about 5 seconds.

Thanks!

I've updated my question with more details. I'm looking for general reference material. — OfLettersAndNumbers, Apr 12 '16 at 23:50
My general reference would be to use an ETL product such as Talend. — Kermit, Apr 13 '16 at 00:31
Thanks for the suggestion, but I want to learn more about the Spark to Vertica connection and porting of data. — OfLettersAndNumbers, Apr 13 '16 at 00:42
There is no Spark to Vertica connection. Unless you know what you're doing, you're not going to run Spark standalone. — Kermit, Apr 13 '16 at 01:20

score 0 · Answer 1 · answered Apr 13 '16 at 03:05

Have you tried HPE's Big Data Marketplace, specifically the HPE Vertica Connector For Apache Spark? You'll need to create an account to download the file, but there's no cost associated with creating an account. The documentation includes a Scala example of writing a Spark data frame to a Vertica table.

How does writing data from spark to vertica work?

1 Answers1