0

I have tried similar suggestions from stack overflow, still issue persist.

I am executing following command from java

public static void main(String[] args) throws Exception {
        try {
            String line;
            //String[] commandToExecute = {"sh" , "MultiDel_FI.sh", " < abcd.log 2>&1 &"};
            String[] commandToExecute = {"sh" , "MultiDel_FI.sh"};


            Process p = Runtime.getRuntime().exec( commandToExecute );
            p.waitFor();
           BufferedReader reader = new BufferedReader(
                    new InputStreamReader(p.getInputStream()));
            writer = new FileWriter("redirect.log");
            while ((line = reader.readLine()) != null) {
                writer.write(line);
          
        }
            writer.close();
            p.waitFor();
            p.destroy();
        catch (Exception e) {
            // 
        }

MultiDel_FI.sh file is placed on edge node with following content

hadoop fs -rm -R -skipTrash 'hdfs://<path>/abc_767' 'hdfs://<path>/abc_768' 'hdfs://<path>/abc_769' ...........many more records

when I run the code it successfully removes the folders from HDFS location. I need to track the output of the command to identify which folder got deleted successfully and which one is not. I tried various options with bin/sh ,writing inputstream to file, it generates empty file. Any suggestions please?

direct unix command properly redirects output.

sh MultiDel_FI.sh > abcd.log 2>&1
user7220859
  • 113
  • 2
  • 2
  • 9
  • https://docs.oracle.com/en/java/javase/16/docs/api/java.base/java/lang/ProcessBuilder.html#redirectOutput(java.io.File) is what you need. Also redirect error – g00se Apr 21 '22 at 12:17
  • thanks, now it is working for sh file, where we are deleting HDFS folders and it redirecting which folders are deleted and which one not found. When I tried same for redirecting beeline .hql file, it does redirect output to file but, redirected file contains lots of info. is there anyway to just log specific messages? – user7220859 Apr 22 '22 at 05:13
  • If you ran beeline via a shell, you could pipe the output through `grep` or `sed` and clean it up – g00se Apr 22 '22 at 09:48
  • I run beeline from java.lang.ProcessBuilder. It repeats at least 10 lines of INFO messages for each partition drop. If it couldn't find the partition id in table still it logs 9 INFO messages. – user7220859 Apr 25 '22 at 09:16
  • I know you do. What I mean is if you run it through `ProcessBuilder` *via* a shell. Something like `String cmd = { "sh", "MultiDel_FI.sh", "|", "sed", "/INFO/d" };` – g00se Apr 25 '22 at 09:38
  • Or maybe `String cmd = { "sh", "MultiDel_FI.sh", "| sed", "/INFO/d" };` But you'd be better off controlling the logging properly – g00se Apr 25 '22 at 09:45

1 Answers1

0
public static void main(String[] args) throws IOException {
        //String[] commandToExecute = {"sh" , "MultiDel_FI.sh"};
        String[] commandToExecute = {"beeline", "--force=true", "-u", "jdbc:hive2://<host>:<port>/<hive database>;ssl=true;user=<user>;password=<pw>", "-f", "ppp1.hql"};
        ProcessBuilder builder = new ProcessBuilder(commandToExecute);
        builder.redirectError(new File("out.txt"));
        builder.redirectOutput(new File("out.txt"));
         builder.start();
    }
user7220859
  • 113
  • 2
  • 2
  • 9