A library for reading and writing Avro data from Spark SQL.
The GitHub page is here.
Questions tagged [spark-avro]
227 questions
-1
votes
1 answer
Value Type is binary after Spark Dataset mapGroups operation even return a String in the function
Environment:
Spark version: 2.3.0
Run Mode: Local
Java version: Java 8
The spark application trys to do the following
1) Convert input data into a Dataset[GenericRecord]
2) Group by the key propery of the GenericRecord
3) Using mapGroups after…
DeepNightTwo
- 4,809
- 8
- 46
- 60
-6
votes
2 answers
design- Can Kafka Producer written as Spark-job?
My pipeline is the following:
Source-webservices ---> Kafka Producer --> topics --> sparkJobs --> hdfs/hive
I have two design-related questions:
I need to pull the data from DataSourceAPIs(web service URLs) and push on to the topics.
If I use…
BdEngineer
- 2,929
- 4
- 49
- 85