spark read xml database record as xml inputStream instead load from file path

Asked Dec 25 '18 at 14:54

Active Dec 25 '18 at 15:33

Viewed 244 times

From spark document

load(): DataFrame
load(path: String): DataFrame
load(paths: String*): DataFrame

I defined function which read a xml record

 def ExtractData(RecID: String,table:String)={
    val spark = SparkSession.
      builder.master("local[*]")
     .appName("Insight Application Big Data")
     .getOrCreate()

    val df = spark.read
      .format("com.databricks.spark.xml")
      .option("rowTag", "row")
      .load("src/main/resources/in/FBNK_CATEG_ENTRY.xml")//pass the xml record here
  }

How can I do if I want to pass the record which is in xml format to .load function, instead of loading from file, something like .load(inputStreamXML)

edited Dec 25 '18 at 15:33

asked Dec 25 '18 at 14:54

tree em

20,379
30
92
130

Write the InputStream to a temporary file... Otherwise, you need Spark Streaming and I don't think Spark Streaming handles multi-line XML very well – OneCricketeer Dec 25 '18 at 14:58
Do you mean read the all records from db then write it to file, then use spark to read of each file? (my db contains more than 9M records.) – tree em Dec 25 '18 at 15:01
If you're reading from a database, then I would suggest using JDBC reader... Your question appears to be reading a single file, however – OneCricketeer Dec 25 '18 at 15:09
thanks can you help to provide an answer with sample using jdbc. – tree em Dec 25 '18 at 15:11
What's wrong with what you posted? Other than you shouldn't be using `.collect()` and `foreach(println)`? – OneCricketeer Dec 25 '18 at 15:32
I mean how we can read using jdbc , can you give me some samples. – tree em Dec 25 '18 at 15:34
1

The only example I have would be the ones on the Spark website... – OneCricketeer Dec 25 '18 at 15:52

spark read xml database record as xml inputStream instead load from file path

0 Answers0