Convert Seq to Pipe in Scalding

Question

Context: I'm reading in a file where multiple fields are a list of IDs. I need to convert these fields into a Pipe to join them with other Pipes.

What I have tried:

val otherPipe = pipe
     .project('fieldIwant)
     .map { p: Pipe => p.toString.split(",") } // converts pipe -> array
     .unique

score 1 · Accepted Answer · answered May 27 '16 at 23:23

1

You can get a TypedPipe from a collection using:

TypedPipe.from(Seq(1, 2, 3, 4, 555, 3))

If you need to fall back to FieldsAPI (which is deprecated), you can do it using

TypedPipe.from(Seq(1, 2, 3, 4, 555, 3)).toPipe('fieldName)

answered May 27 '16 at 23:23

Dan Osipov

1 Answers1