0

I'm writing a script that retroactively performs an ETL process we previously missed. The bit I'm having trouble with is this piece of code:

feed_string = java_home + '/java -Xmx6000m -jar ' + script_dir + '/Proj/Feeds.jar -o ' + feed + '_ad_hoc_load_' + date + '.csv -s FileSender -d ' + date + ' -f ' + feed + ' -r ' + script_dir + '/Feeds --config {root}/config || { echo \'' + feed + ' process FAILED...\' ; log $AUTO_JOB_NAME "Failure" 1 "' + feed + ' failed..."; exit 1; }'
print('Executing shell command:')
print(feed_string)
print('')
feed_string = str.split(feed_string, ' ')
proc = subprocess.Popen(feed_string)
proc.wait()
print proc.returncode

When I test this I use 3 dates for files; 2 of which exist, one of which doesn't. The first two dates run without error, transform the data and the Python code prints 0 in my standard out. The final date however does not have an accompanying file and as a result, my logs print out a stack trace saying java.io.FileNotFoundException: /path/to/file/file_20170908.data (No such file or directory) - as expected.

The problem is that the Python code then prints out another 0 despite the fact it throws a FileNotFoundExecption. Why is this?

lapinkoira
  • 8,320
  • 9
  • 51
  • 94
Nanor
  • 2,400
  • 6
  • 35
  • 66
  • If you manually run that command can you capture the returncode different than 0? (without using python) – lapinkoira Sep 25 '17 at 09:09
  • How do I get a return code? Executing the command just prints out the logs and stack trace. – Nanor Sep 25 '17 at 09:36
  • Are you on linux? Run a command and then run echo $? it will return 0 or 127 for example. To test it for example you can type "asdafg", it will return "command not found" then tthe echo $? will return 127, or if you type "ps" it will show the processes list and will return 0 – lapinkoira Sep 25 '17 at 09:42
  • Ah, running the command in this case returns `-bash-4.1$ echo $? 0` – Nanor Sep 25 '17 at 09:43
  • Yeah, so python just print that, it doesnt do any magic on top of that, if the java program itself doesnt return an error code you will have for example to parse the proc output instead checking the error – lapinkoira Sep 25 '17 at 09:44
  • That's annoying, I would have thought throwing any `Exception` would result in a non-zero return code. Is that a result of Java itself or a problem with our Java code not explicitly returning a non-zero? – Nanor Sep 25 '17 at 09:46
  • You should show the java code, are you for example doing something like this? System.exit(0); it will always return 0 – lapinkoira Sep 25 '17 at 09:48
  • Check this https://stackoverflow.com/questions/6775802/return-value-from-a-java-code – lapinkoira Sep 25 '17 at 09:49

1 Answers1

0

You could try something like this to parse the output

from subprocess import Popen, PIPE

feed_string = java_home + '/java -Xmx6000m -jar ' + script_dir + '/Proj/Feeds.jar -o ' + feed + '_ad_hoc_load_' + date + '.csv -s FileSender -d ' + date + ' -f ' + feed + ' -r ' + script_dir + '/Feeds --config {root}/config || { echo \'' + feed + ' process FAILED...\' ; log $AUTO_JOB_NAME "Failure" 1 "' + feed + ' failed..."; exit 1; }'
p = Popen(feed_string, stderr=PIPE, stdout=PIPE, shell=True)
output, errors = p.communicate()
print [p.returncode, errors, output]
# Check here what's inside errors or output
lapinkoira
  • 8,320
  • 9
  • 51
  • 94