0

SparkR Column provides a long list of useful methods for example 'isNull' but in sparkR I have an issue using them. I run sparkR in R like this

cd /home/ole/R/spark-1.4.0 ./bin/sparkR --packages com.databricks:spark-csv_2.10:1.0.3 sqlContext

When I for example type this u=c() isNull(u) I get this message Error in (function (classes, fdef, mtable) : unable to find an inherited method for function ‘isNull’ for signature ‘"NULL"

f_puras
  • 2,521
  • 4
  • 33
  • 38
Ole Petersen
  • 670
  • 9
  • 21

1 Answers1

2

That is because isNull expects a column of a DataFrame and not a vector. It checks wether entries are NULL, it works as follows:

a   <- createDataFrame(sqlContext,data.frame(b=c("a","b",NA,"c"),c=c(1,2,3,4)))
a$d <- isNull(a$b)
collect(a)

It also returns a (logical) column, that's why I appended it to the DataFrame. You will notice however that SparkR did not store the NA as NULL, so all logicals are FALSE, but you already see how the function is working.

Wannes Rosiers
  • 1,680
  • 1
  • 12
  • 18