I am unable to execute a pig command from Java. The command is to create a relation, load an input file in the relation and then store it into a file in hdfs path. Following is the error I am getting:
15/09/23 05:02:50 WARN mapReduceLayer.Launcher: There is no log file to write to.
15/09/23 05:02:50 ERROR mapReduceLayer.Launcher: Backend error message during job submission org.apache.pig.backend.executionengine.ExecException: ERROR 2100: file:/<username>/abhi/myinput.txt does not exist.
at org.apache.pig.backend.executionengine.PigSlicer.validate(PigSlicer.java:126)
at org.apache.pig.impl.io.ValidatingInputFileSpec.validate(ValidatingInputFileSpec.java:59)
at org.apache.pig.impl.io.ValidatingInputFileSpec.<init>(ValidatingInputFileSpec.java:44)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:230)
at org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:810)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:781)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
at java.lang.Thread.run(Thread.java:745)
15/09/23 05:02:50 ERROR mapReduceLayer.MapReduceLauncher: Failed to produce result in: "file:/user/<username>/abhi"
15/09/23 05:02:50 INFO mapReduceLayer.MapReduceLauncher: Failed! And below is the code package oozie;
This is the code:
import java.io.IOException;
import java.util.Properties;
import org.apache.pig.ExecType;
import org.apache.pig.PigServer;
import org.apache.pig.impl.PigContext;
public class Oozietest1 {
public static void main(String[] args) {
try {
Properties pr = new Properties();
pr.setProperty("fs.defaultFS", "hdfs://<ip address of the resource manager>/");
pr.setProperty("mapreduce.framework.name", "yarn");
pr.setProperty("yarn.resourcemanager.address", "< address>:<port>");
PigContext pc=new PigContext(ExecType.MAPREDUCE, pr);
PigServer pigServer = new PigServer(pc);
System.out.println(pigServer);
runMyQuery(pigServer, "hdfs://user/<username>/abhi/myinput.txt");
} catch (IOException e) {
e.printStackTrace();
}
}
public static void runMyQuery(PigServer pigServer, String inputFile) throws IOException {
pigServer.registerQuery("A = load '" + inputFile + "' using PigStorage(',') as (name);");
pigServer.registerQuery("STORE A INTO '/user/<username>/abhi/' USING PigStorage (',');");
}
}
In brief the file is always attempted to be read from local file system, whereas it should have been read from hdfs. Please let me know if you need any additional information.
