Working with Livy on the Hortonworks Sandbox we found the problem that on some cases the Livy API provide a result for the statement before the actual work has ended as indicated by the Spark monitoring tool. In some cases Livy provide a result in just less than 5 seconds for a job with normal execution time of more than 10 minutes. If we repeat the same task with the same initial conditions more than once on some cases we have seen this problem happen. Any idea?
Asked
Active
Viewed 67 times
1 Answers
0
Just check if you are using the same Livy session for multiple jobs or not. If same session is used then intermediate Spark DataSet is getting cached and results are retrieved faster.
dassum
- 4,727
- 2
- 25
- 38
-
This is an older question but I will test if this is the case – yeiniel Jan 19 '18 at 21:44