0

I am want to create a parquet file from payload in CentOs. Here is what I did. 

parquetDataSet.write().mode(SaveMode.Append).parquet(tempFile.getAbsolutePath());

here is the dependency I used

<dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.12</artifactId>
        <version>2.4.1</version>
    </dependency>

Help me with the following is the error. 

 Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Handler dispatch failed; nested exception is java.util.ServiceConfigurationError: org.apache.spark.sql.sources.DataSourceRegister: Provider org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat could not be instantiated] with root cause



java.lang.ClassNotFoundException: org.apache.parquet.hadoop.ParquetOutputFormat$JobSummaryLevel
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[na:1.8.0_212]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[na:1.8.0_212]
        at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:92) ~[app.jar:0.0.1-SNAPSHOT]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[na:1.8.0_212]
        at java.lang.Class.getDeclaredConstructors0(Native Method) ~[na:1.8.0_212]
        at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671) ~[na:1.8.0_212]
        at java.lang.Class.getConstructor0(Class.java:3075) ~[na:1.8.0_212]
        at java.lang.Class.newInstance(Class.java:412) ~[na:1.8.0_212]
Sowmya
  • 107
  • 4
  • 13

1 Answers1

1

add the below dependency 

<dependency>
    <groupId>org.apache.parquet</groupId>
    <artifactId>parquet-hadoop</artifactId>
    <version>1.11.0</version>
</dependency>

you can get rid of ClassNotFoundException ParquetOutputFormat$JobSummaryLevel

I hope this helps.

jose praveen
  • 1,298
  • 2
  • 10
  • 17
  • But this file is not found now: org.apache.parquet.hadoop.metadata.CompressionCodecName; In 1.8.0 and lower versions, compressioncodecname is available but parquetfileformat is unavailable. I want both of them in the same repository. – Sowmya Mar 11 '20 at 08:23