Integrating Spark MLLib algorithm to H2O ai using Sparkling water

Question

I am trying to integrate Collaborative algorithm in Spark MLLib with H2o Ai using Sparkling water for product recommendation. I followed this link

http://spark.apache.org/docs/latest/mllib-collaborative-filtering.html

and updated code as in below

System.setProperty("hadoop.home.dir", "D:\\backup\\lib\\winutils")
    val conf = new SparkConf()
      .setAppName("Spark-InputFile processor")
      .setMaster("local")

    val sc = new SparkContext(conf)

    val inputFile = "src/main/resources/test.data"

    val data = sc.textFile(inputFile)

    val ratings = data.map(x=>{
      val mapper = x.split(",")
      Rating(mapper(0).toInt,mapper(1).toInt,mapper(2).toDouble)
    })
    // Build the recommendation model using ALS
    val rank = 10
    val numIterations = 10
    val model = ALS.train(ratings, rank, numIterations, 0.01)



    // Save and load model
    model.save(sc, "target/tmp/myCollaborativeFilter")
    val sameModel = MatrixFactorizationModel.load(sc, "target/tmp/myCollaborativeFilter")

    val modelRdd = sameModel.recommendProductsForUsers(100)

    implicit val sqlContext = SparkSession.builder().getOrCreate().sqlContext
    import sqlContext.implicits._

    val modelDf = modelRdd.toDF("Rdd","Rdd1")




    @transient val hc = H2OContext.getOrCreate(sc)

    val h2oframe:H2OFrame = hc.asH2OFrame(modelDf)

When I run the code in Intellij I am getting the below error

Exception in thread "main" java.util.NoSuchElementException: key not found: StructType(StructField(user,IntegerType,false), StructField(product,IntegerType,false), StructField(rating,DoubleType,false))
    at scala.collection.MapLike$class.default(MapLike.scala:228)
    at scala.collection.AbstractMap.default(Map.scala:59)
    at scala.collection.MapLike$class.apply(MapLike.scala:141)
    at scala.collection.AbstractMap.apply(Map.scala:59)
    at org.apache.spark.h2o.utils.ReflectionUtils$.vecTypeFor(ReflectionUtils.scala:132)
    at org.apache.spark.h2o.converters.SparkDataFrameConverter$$anonfun$3.apply(SparkDataFrameConverter.scala:68)
    at org.apache.spark.h2o.converters.SparkDataFrameConverter$$anonfun$3.apply(SparkDataFrameConverter.scala:68)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
    at scala.collection.Iterator$class.foreach(Iterator.scala:893)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
    at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
    at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
    at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
    at scala.collection.AbstractTraversable.map(Traversable.scala:104)
    at org.apache.spark.h2o.converters.SparkDataFrameConverter$.toH2OFrame(SparkDataFrameConverter.scala:68)
    at org.apache.spark.h2o.H2OContext.asH2OFrame(H2OContext.scala:132)
    at org.apache.spark.h2o.H2OContext.asH2OFrame(H2OContext.scala:130)
    at com.poc.sample.RecommendataionAlgo$.main(RecommendataionAlgo.scala:54)
    at com.poc.sample.RecommendataionAlgo.main(RecommendataionAlgo.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)

How can I solve this error?

Thanks in advance.

score 1 · Accepted Answer · answered Feb 18 '17 at 07:44

1

modelRdd will be of type Tuple2<Object, Rating> (or equivalent in Scala), Rating isn't a type we (Sparkling Water) provide automatic conversion for (it's not a String, Double, Float etc. nor does it implement Product). We definitely need to throw a more meaningful error message there.

To fix this instead of making a DataFrame with Object, Rating with modelRdd.toDF("Rdd","Rdd1") you can map it into a DF with 4 columns Object, user, product, rating and then use hc.asH2OFrame().

answered Feb 18 '17 at 07:44

Mateusz Dymczyk

14,969
10
59
94

That solves my first question. Any solid links for complete example to get start with Sparkling water end to end? – mvg Feb 20 '17 at 13:23
@mvg I know our documentation is a bit behind - we're working on updating it to 2.x but I think it still should be doable to get a full working example by following our Github README and standard spark practices. Which parts do you find challenging? – Mateusz Dymczyk Feb 21 '17 at 03:27
I took this code https://github.com/h2oai/sparkling-water/blob/master/examples/src/main/scala/org/apache/spark/examples/h2o/AmazonFineFood.scala and created jar file out of it and submitted it to Spark-Submit and it starts H2o flow. I opened it in browser and stuck there, because it more looks like a config tool. – mvg Feb 21 '17 at 05:13
@mvg could you post a new question with a bit more details (how you created the jar, what's your spark submit command etc). – Mateusz Dymczyk Feb 21 '17 at 05:20
sure will close this question and open a new one. Thanks !! – mvg Feb 21 '17 at 05:38

Integrating Spark MLLib algorithm to H2O ai using Sparkling water

1 Answers1