0

I truly need your assistance, and I hope I do not sound too harsh when I say that I am not looking to get into debugging my code. I am hoping and/or think that it is something else besides my code that perhaps I do not expect to happen with spark.

First, let me give you some history and a timeline so I can put it on perspective (or you can skip down to the question below ;)).

I've developed a piece of code on python that uses mllib back in Oct 2015 by I using spark 1.5.1 and it worked. Stayed using this package for awhile, until Aug 2016. For whatever reason, my code was not working anymore, even though I was using the same input files that I used between Oct 2015 to Aug 2016. My fix was to upgrade to spark 2.0.0. Ran same code with same input files and it started working again. [Now] Two months later I need to do a refresh and guess what, it is not working again. I went back to the same input files that I have used on spark 1.5.1 and when it successfully ran in 2.0.0 it still doesn't work.

So, my question is... Is spark doing some update or something in the background that I am not aware of and should be aware of? If it is, how do I turn it off or can I turn it off? If this behavior is not expected (which I don't think it is), why it might be occurring?

Thanks for your help in advance.

blacknred0
  • 36
  • 3
  • Most probable explanation: Either your code or configuration is incorrect in a way which makes it fail nondeterministically. Voting to close because without both all we can do is guess. –  Oct 26 '16 at 01:47
  • It would make sense if I made changes, but what doesn't is if I haven't made any changes to it. Why would it break months later if I use the same inputs and code that worked before? BTW - I agree that for the most extent would be guessing without debugging this, but wanted to get a sense to see if others were experiencing this or not. – blacknred0 Oct 26 '16 at 13:08

1 Answers1

0

After further testing, this post solved the problem -> Spark gives a StackOverflowError when training using ALS. I hope this a long-term solution rather than a patch.

Community
  • 1
  • 1
blacknred0
  • 36
  • 3