scala type mismatch osmosis

Question

I try to get the example from this open book about map matching to work. I am using scala 2.10 and spark 2.0.2.

Unfortunatelly the syntax and functions have changed.

case class NodeEntry(nodeId: Long, latitude: Double, longitude: Double, tags: Array[String])

val nodeDS = nodes.map{node => 
NodeEntry(node.getId, 
   node.getLatitude, 
   node.getLongitude, 
   node.getTags.map(_.getValue).toArray
)}.toDS.cache

I get the Error "value map is not a member of java.util.List[org.openstreetmap.osmosis.core.domain.v0_6.WayNode]"

and "value toDS is not a member of scala.collection.mutable.ArrayBuffer[Nothing] possible cause: maybe a semicolon is missing before `value toDS'?"

I tried to change the nodes.map to the following:

val nodeDS = nodes.map { node => 
  NodeEntry(node.getId, 
      node.getLatitude, 
      node.getLongitude, 
      node.getTags.toArray() 
  )}

But then I get this Error: type mismatch; found : Array[Object] required: Array[String] Note: Object >: String, but class Array is invariant in type T. You may wish to investigate a wildcard type such as _ >: String. (SLS 3.2.10)

score 1 · Answer 1 · edited Dec 17 '16 at 13:16

1

Ah I got it. After the suggested Import the following worked:

val nodeDS = nodes.map { node => 
  NodeEntry(node.getId, 
      node.getLatitude, 
      node.getLongitude, 
      node.getTags.map(_.getValue).toArray
  )}

edited Dec 17 '16 at 13:16

Termininja

6,620
12
48
49

answered Dec 08 '16 at 11:43

Simon He

47
7

` val sqlContext = new SQLContext(sc) import sqlContext.implicits._` must be in your code, to make the .toDS.cache part work – Simon He Dec 08 '16 at 13:36

score 0 · Accepted Answer · answered Dec 08 '16 at 10:08

map is not available on Java collections inorder to make it available import JavaConversions

import scala.collection.JavaConversions._

Example

Scala REPL

scala> val l = new java.util.ArrayList[String]()
l: java.util.ArrayList[String] = []

scala> l.add("scala")
res0: Boolean = true

scala> l.add("haskell")
res1: Boolean = true

scala> l.map(x => x.reverse)
<console>:13: error: value map is not a member of java.util.ArrayList[String]
       l.map(x => x.reverse)
     ^

scala> import scala.collection.JavaConversions._
import scala.collection.JavaConversions._

scala> l.map(x => x.reverse)
res3: scala.collection.mutable.Buffer[String] = ArrayBuffer(alacs, lleksah)

Notice that after importing map is now available on the java ArrayList

`type mismatch; found : Iterable[String] required: Array[String]` is the error I get for the following code. It marks the "_.getValue" here `val nodeDS = nodes.map { node => NodeEntry(node.getId, node.getLatitude, node.getLongitude, node.getTags.map(_.getValue) )}` — Simon He, Dec 08 '16 at 11:38

scala type mismatch osmosis

2 Answers2