1

I have created IBM BigInsights service with hadoop cluster of 5 nodes(including Apache Spark with SparkR). I trying to use SparkR to connect cloudant db and get some data and do some processing.

SparkR job(R script) submit using spark-submit fails in BigInsights Hadoop cluster. I have created SparkR script and ran the following code, 

-bash-4.1$ spark-submit --master local[2] test_sparkr.R
16/08/07 17:43:40 WARN SparkConf: The configuration key 'spark.yarn.applicationMaster.waitTries' has been deprecated as of Spark 1.3 and and may be removed in the future. Please use the new key 'spark.yarn.am.waitTime' instead.
Error: could not find function "sparkR.init"
Execution halted
-bash-4.1$

Content of test_sparkr.R file is:

# Creating SparkConext and connecting to Cloudant DB
sc <- sparkR.init(sparkEnv = list("cloudant.host"="<<cloudant-host-name>>","<<><<cloudant-user-name>>>","cloudant.password"="<<cloudant-password>>", "jsonstore.rdd.schemaSampleSize"="-1"))

# Database to be connected to extract the data
database <- "testdata"
# Creating Spark SQL Context
sqlContext <- sparkRSQL.init(sc)
# Creating DataFrame for the "testdata" Cloudant DB
testDataDF <- read.df(sqlContext, database, header='true', source = "com.cloudant.spark",inferSchema='true')

How to install the spark-cloudant connector in IBM BigInsights and resolve the issue. Kindly do the needful. Help would be much appreciated.

Pari Margu
  • 209
  • 3
  • 10

1 Answers1

0

I believe that the spark-cloudant connector isn’t for R yet.

Hopefully I can update this answer when it is!

JasonSmith
  • 72,674
  • 22
  • 123
  • 149
  • spark-cloudant connector is works with SparkR notebook(R Technical Preview) in spark service in Bluemix, hence I thought it would be available in SparkR in IBM BigInsights cluster. Is there any way I can put those jars(spark-cloudant connector) and try to connect with cloudant database from SparkR? Thank you. – Pari Margu Aug 09 '16 at 12:34
  • Hi @JasonSmith, is it possible to submit my SparkR script(job) using spark-submit command? – Pari Margu Aug 09 '16 at 12:41