1

I'm trying to build a simple WordCount jar project which utilizes Hadoop-lzo library but cannot seem to get the following command to work, even though the class I'm referencing is within hadoop classpath:

$ javac -cp `hadoop classpath` *.java

LzoWordCount.java:76: cannot find symbol
symbol  : class LzoTextInputFormat
location: class LzoWordCount
      job.setInputFormatClass(LzoTextInputFormat.class);
                              ^
1 error

Any ideas?

Carl Sagan
  • 982
  • 1
  • 13
  • 34
  • I don't think that `LzoTextInputFormat` is part of the hadoop distro (which version are you using?) It appears to be part of a google code project named hadoop-gpl-compression - https://code.google.com/a/apache-extras.org/p/hadoop-gpl-compression/ – Chris White Aug 02 '13 at 01:26
  • I am using Hadoop 2.0.0-cdh4.3.0. How would I include the correct .java or .class for my code to build? – Carl Sagan Aug 05 '13 at 07:15
  • LzotextInputFormat isn't listed in the Javadocs for CDH4u4: http://archive.cloudera.com/cdh4/cdh/4/hadoop/api/ - Where did you acquire the class from? – Chris White Aug 06 '13 at 01:47
  • I followed: http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM4Ent/latest/Cloudera-Manager-Installation-Guide/cmig_install_LZO_Compression.html - and added the Parcel – Carl Sagan Aug 06 '13 at 19:06

1 Answers1

0

I assume you have correctly installed your LZO libraries (you should have libgplcompression.so in your lib/natives/Linux**-**/ and the jar file in your lib/ folder)

Since you have them the correct class should be LzoDeprecatedTextInputFormat.class or LzoTextInputFormat.class depending on wich API you use (According to your post you are using it right Job with LzoTextInputFormat).

So your problem could be in your java.library.path, wich should include path to your jar file. You can set it up in your .bash_profile or in you bin/hadoop file. hope that helps.

Brugere
  • 436
  • 4
  • 16
  • I have tried directly adding the LzoTextInputFormat.class file into my project lib/ folder as well as actually including the LzoTextInputFormat.java source in my src/ folder, but neither works. – Carl Sagan Aug 05 '13 at 07:16
  • Sorry if I failed explaining myself but I meant include your `.jar` file (something like `hadoop-lzo-***.jar`) in your `java.library.path`. Because if you open this jar with vim you will see all the classes you need in it. I think there is also a way using the `-Djava.library.path=path/to/your/lzo/jar` in your programm command line. Hope that better helps. – Brugere Aug 05 '13 at 15:24