A library for handling CSV files in Apache Spark.
External links:
Questions tagged [spark-csv]
139 questions
-1
votes
1 answer
How to detect change in CSV file schema in Spark
If there is change in schema in my incoming CSV file, how can we handle this in spark?
Suppose on Day-1,
I got a csv file with schema and data as below,
FirstName LastName Age
Sagar Patro 26
Akash Nayak 22
Amar Kumar 18
And on…
Sagar patro
- 115
- 2
- 11
-1
votes
1 answer
pyspark tsv duplicate columns
I am trying to read a tsv file in pyspark using spark_csv package. My spark version is 1.6.3. In my dataset, two columns have same name. I am using following code to read the data.
temp=sqlContext.read.load(data_file,
…
mc29
- 85
- 2
- 10
-1
votes
2 answers
Replacing null value with blank value in spark data frame not working
I have two dataset,
dataset 1 is…
Sudarshan kumar
- 1,503
- 4
- 36
- 83
-1
votes
1 answer
how to write "DATE" data type in spark-csv
i have data similar to this.
no,full_name,Ocupation,DOB
1,Eugene Perry,Physical Therapy Assistant,11/22/1972
2,Scott Evans,Account Executive,8/25/1961
3,Harold Porter,Community Outreach Specialist,4/5/1986
4,Roger Turner,Account…
sriramkumar
- 144
- 2
- 15