1

My data format uses \0 instead of new line. So default hadoop textLine reader dosn't work. How can I configure it to read lines separated by special character?

If it is impossible to configure LineReader, Maybe it is possible to apply specic stream processor(tr "\0" "\n") not sure how to do this.

yura
  • 14,489
  • 21
  • 77
  • 126

3 Answers3

2

There is a "textinputformat.record.delimiter" configuration property for that purpose. You can change default EOL ("\n") delimiter by changing this property value to "\0".

For more information, go here: http://amalgjose.wordpress.com/2013/05/27/custom-text-input-format-record-delimiter-for-hadoop

There is also a similar question for about changing the default delimiter in spark, which may be useful too: Setting textinputformat.record.delimiter in spark

Community
  • 1
  • 1
k-sever
  • 975
  • 10
  • 13
1

You can write your own InputFormat class that splits data on \0 instead of \n. For a walkthrough on how to do that, check here: http://developer.yahoo.com/hadoop/tutorial/module5.html#fileformat

The gist of it is that you need to subclass the default InputFormat class, or any of its subclasses, and define your own RecordReader with custom rules. For more on that, you can refer to the InputFormat documentation.

HypnoticSheep
  • 843
  • 1
  • 7
  • 16
0

How about, would using a TextDelimited scheme work? http://docs.cascading.org/cascading/1.2/javadoc/cascading/scheme/TextDelimited.html

That avoids having to write your own InputFormat, etc.

Examples of text delimited are in https://github.com/Cascading/Impatient/wiki/Part-2

Paco
  • 602
  • 1
  • 9
  • 19