we are using a Apache Livy server to communicate with a spark cluster. We have noticed that some statements when executed error out and afterwards livy session becomes unusable. For example we are using fbprophet for timeseries models. fbprophet and pystan produce some console output. As soon as this output is logged in session the statement gives error. Although the job continues to execute on the cluster. And afterwards no statement runs correctly. Output after running statement with fbprophet code
Unrecognized token 'Iter': was expecting 'null', 'true', 'false' or NaN
at [Source: Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes ; line: 1, column: 9]
Similarly while creating docker image using mlflow build-docker command, it also produces output about the image creation process. Again as soon as this output is generated the session errors out Following is the output for build-docker
Unexpected character ('/' (code 47)): maybe a (non-standard) comment? (not recognized as one since ``````Feature 'ALLOW_COMMENTS' not enabled for parser)
at [Source: /tmp/tmpq6gz1q3t/; line: 1, column: 2]
Main issue is that the session becomes unusable after such statements. Has anyone faced such issues with Apache livy?
One resolution that I can think is to suppress the output from pystan and docker processes, But I could not figure that out. How can we suppress the output from being logged into livy session?
