Scalding + LZO +Protobuf

Question

Are there any pointers to get Scalding to work with LZO Protobuf data on HDFS?

I am trying to read files that are stored in binary Protobuf and compressed in LZO using Scalding. Can we use Elephantbird to read those files? Any pointers will be appreciated!

I have looked at the LzoTraits and LzoProtobufScheme? But I am not sure how I should be using it to read the data? Any examples would be great!

score 1 · Answer 1 · answered Feb 14 '14 at 20:27

1

Here is an example:

case class SomeProto() extends FixedPathSource("/my/greatData/*")
  with LzoProtobuf[MyProtoClassHere] {
    override def column = classOf[MyProtoClassHere]
}

You can mix with other types of abstract base Sources (like TimePathedSource, or MostRecentGoodSource) in a similar way. You can mix in with LocalTapSource if you want to use the Hadoop-inside-cascading-local trick (if you don't run in cascading local mode, you don't need this).

answered Feb 14 '14 at 20:27

Oscar Boykin

1,974
2
11
16

Thanks, Will the same class also work for writing out Lzo compressed protobuf files? – thinker25 Mar 03 '14 at 21:57
Another quick followup question. How do I read binary protobuf files that are not lzo compressed? – thinker25 Apr 18 '14 at 01:27

Scalding + LZO +Protobuf

1 Answers1