I'm writing a wrapper to run MrJob jobs with, and it's working quite well but I'd like to be able to deliver the Python stack trace from the job if it throws an exception. Originally, when something went wrong (i.e. an assert in the job code fails) I got the mrjob exception instead, which was not helpful.
On a Google Groups thread I found the function set_up_logging(), which works quite well. I use it as follows:
linecount_job = LineCount(args=['-r', 'hadoop', target_filename])
linecount_job.set_up_logging(stream=sys.stdout)
with linecount_job.make_runner() as runner:
...
and it dumps a bunch of feedback, including the Python stack trace from the job, to stdout.
Is there a way to capture just the stack trace (the goal is to display it in a web app)? My plan right now is to scrape the data for stuff that looks like a stack trace, which is ugly but would probably work.
I'm also concerned that I can't find documentation for this function at all. It seems like it should be here. Is this function deprecated? Is there a better way to do what I'm doing?
