How to write csv file in apache spark using SparkR?

Question

I am able to load data Successfully using following commands

sc = sparkR.init(master = 'local', sparkPackages = 'com.databricks:spark-csv_2.11:1.4.0')
sqlContext <- sparkRSQL.init(sc)
ss <- read.df(sqlContext, '/home/anmol/Downloads/Rgraphics/dataSets/states.csv', source = "com.databricks.spark.csv", inferSchema = "true")
head(ss)

I have tried the following command

write.df(df, '/home/anmol/faithfull.csv', source = 'com.databricks.spark.csv', 'overwrite')

but it gives following error

16/06/10 18:28:26 ERROR RBackendHandler: save on 261 failed Error in invokeJava(isStatic = FALSE, objId$id, methodName, ...) : java.lang.NoClassDefFoundError: Could not initialize class com.databricks.spark.csv.util.CompressionCodecs$ at com.databricks.spark.csv.DefaultSource.createRelation(DefaultSource.scala:189) at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:222) at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:148) at org.apache.spark.sql.DataFrame.save(DataFrame.scala:2027) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.api.r.RBackendHandler.handleMethodCall(RBackendHandler.scala:141) at org.apache.spark.api.r.RBackendHandler.channelRead0(RBackendHandler.scala:86) at org.apache.spark.api.r.RBackendHandler.channelRead0(RBackendHandler.scala:38) at io.netty.channel.

definitely a version error like zero323 pointed out. use spark-csv_2.10 instead — eliasah, Jun 10 '16 at 13:37
but you are using com.databricks:spark-csv_**2.11**:1.4.0 that's compiled for scala 2.11 — eliasah, Jun 10 '16 at 13:42
I have instaled scala 2.11.7 still it dosen't work same problem able to read not able to write — anmol sandhu, Jun 10 '16 at 14:04
It is not about which version is installed. It is about version which has been used to compile Spark that has to match version of the packages you download. With provided binaries use 2.10 — zero323, Jun 10 '16 at 15:19

score 1 · Accepted Answer · answered Jun 10 '16 at 16:13

the problem was the version that is being used to compile my apache spark it was 2.10 so i used

sc <- sparkR.init(master = 'local', sparkPackages = 'com.databricks:spark-csv_2.10:1.4.0')

you can check yours by logging into spark-shell it gives the version of scala while starting up

How to write csv file in apache spark using SparkR?

1 Answers1