1

I have followed the QuickStart Guide at http://predictionio.incubator.apache.org/templates/recommendation/quickstart/

Installed PredictionIO. EventServer works fine. New App created successfully. GET and POST requests work fine. Import the sample-data worked fine.

pio build

worked fine. but when i try to run

pio train

i get this error

[INFO] [Engine] Extracting datasource params...
[INFO] [WorkflowUtils$] No 'name' is found. Default empty String will be used.
[INFO] [Engine] Datasource params: (,DataSourceParams(MyApp1,None))
[INFO] [Engine] Extracting preparator params...
[INFO] [Engine] Preparator params: (,Empty)
[INFO] [Engine] Extracting serving params...
[INFO] [Engine] Serving params: (,Empty)
[WARN] [Utils] Your hostname, damiano-asus resolves to a loopback address: 127.0.0.1; using 10.0.10.150 instead (on interface wlp3s0)
[WARN] [Utils] Set SPARK_LOCAL_IP if you need to bind to another address
[INFO] [Remoting] Starting remoting
[INFO] [Remoting] Remoting started; listening on addresses :[akka.tcp://sparkDriver@10.0.10.150:33231]
[WARN] [MetricsSystem] Using default name DAGScheduler for source because spark.app.id is not set.
[INFO] [Engine$] EngineWorkflow.train
[INFO] [Engine$] DataSource: damiano.company.DataSource@483b0690
[INFO] [Engine$] Preparator: damiano.company.Preparator@fb0a08c
[INFO] [Engine$] AlgorithmList: List(damiano.company.ALSAlgorithm@6a5e167a)
[INFO] [Engine$] Data sanity check is on.
[INFO] [Engine$] damiano.company.TrainingData does not support data sanity check. Skipping check.
[INFO] [Engine$] damiano.company.PreparedData does not support data sanity check. Skipping check.
[WARN] [BLAS] Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS
[WARN] [BLAS] Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS
[WARN] [LAPACK] Failed to load implementation from: com.github.fommil.netlib.NativeSystemLAPACK
[WARN] [LAPACK] Failed to load implementation from: com.github.fommil.netlib.NativeRefLAPACK
[Stage 29:>                                                         (0 + 0) / 4][ERROR] [Executor] Exception in task 1.0 in stage 40.0 (TID 137)
[ERROR] [Executor] Exception in task 0.0 in stage 40.0 (TID 136)
[ERROR] [Executor] Exception in task 3.0 in stage 40.0 (TID 139)
[ERROR] [Executor] Exception in task 2.0 in stage 40.0 (TID 138)
[WARN] [TaskSetManager] Lost task 1.0 in stage 40.0 (TID 137, localhost):     java.lang.StackOverflowError
at     java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2901)
at     java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1700)

i guess it is due to the jvm heap/stack dimension? anyone know how ti fix this? thank you

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Damiano Caprari
  • 73
  • 8
  • Are all those warnings before the crash expected? If not, I'd focus on fixing them first. – Aaron Feb 27 '17 at 09:40
  • If you have fixed your problem please close it, if you have partially fixed your problem please ask for further help from users on stackoverflow. – Paulie Feb 27 '17 at 10:13
  • Please put your solution below as an answer rather than in the question itself – OneCricketeer Feb 27 '17 at 14:06
  • I experienced the exact same problem with the quickstart. Changing the number of iterations to 10 solved the problem for the quickstart. Now I deployed an trained engine with a use case with around 300.000 records with the num iterations to 20 and the exact same setup. –  Apr 12 '17 at 14:09

2 Answers2

1

there is a quick work-around for the problem you can edit the engine.json file to lower the number of iterations
eg: "numIterations": 10, instead of 20

but if you need to increase the memory you can follow the FAQs at the bottom of the quickstart-guide page

http://predictionio.incubator.apache.org/resources/faq/#engine-training

start a spark master with ./sbin/start-master.sh
(if you followed the guide spark is in the .../vendors/spark-1.5.1-bin-hadoop2.6/ folder)
start a spark slave with ./sbin/start-slave.sh spark://<yourlocalhost>:7077
you should be able to see all running at http://localhost:8080

then you can run
pio train -- --master spark://localhost:7077 --driver-memory 16G --executor-memory 24G
i'm able to do 20 iterations with --driver-memory 3G --executor-memory 4G on a laptop

if you still have problems, try to configure the spark-default.conf file to suit your needs
i also had to add a line at the start of the .conf file
spark.mesos.coarse true

Damiano Caprari
  • 73
  • 8
0

Upgrade your spark version to spark-1.6.3-bin-hadoop2.6. I had similar issue what you are facing, upgrading to mentioned version resolved my issue.

Nishikant Nipane
  • 58
  • 10