How to read avro file using pyspark

Question

I am trying to read avro file in jupyter notebook but facing this issue.

Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.avro.AvroFileFormat.DefaultSource

and I can't seem to figure out where how to get this dependency from.

import findspark
findspark.init()

import pyspark
from pyspark.sql import SparkSession
from pyspark.sql.types import *

spark = SparkSession.builder.appName("readavro").master("local").getOrCreate()

result = spark.read.format('com.databricks.spark.avro').load("file:///C:/Downloads/part-r-00000.avro")

Can you accept the answer or explain why it doesn't work? – QuickSilver May 17 '20 at 06:41 — QuickSilver, May 17 '20 at 06:41

QuickSilver · Accepted Answer · 2020-04-23T09:39:14.557

1

Make sure you add org.apache.spark:spark-avro_2.12:2.4.5 jar to your classpath. Since spark-avro module is external, there is no .avro API in DataFrameReader or DataFrameWriter. So try

result = spark.read.format('avro').load("file:///C:/Downloads/part-r-00000.avro")

include the avro dependency

$ bin/spark-shell --packages com.databricks:spark-avro_2.12:2.4.5

edited Apr 23 '20 at 09:39

answered Apr 23 '20 at 09:02

QuickSilver

3,915
2
13
29

It is giving me exception Caused by: java.lang.NoSuchMethodError: org.apache.spark.sql.execution.datasources.FileFormat.$init$(Lorg/apache/spark/sql/execution/datasources/FileFormat;)V – user1298426 Apr 23 '20 at 09:29
@user1298426 updated ans to include avro dependency – QuickSilver Apr 23 '20 at 09:39
I have added spark-avro_2.12:2.4.5 in my $SPARK_HOME/jars. Should it work? because getting the same exception. – user1298426 Apr 23 '20 at 09:46
1

Please check the compatibility of your avro, Spark and scala version it should be same. Also check the scala version of Avro and Spark jars. – QuickSilver Apr 23 '20 at 10:01
this was indeed a versioning issue. – user1298426 May 20 '20 at 06:57

How to read avro file using pyspark

1 Answers1