0

I am using SimilarProduct template of prediction.io Inserted 16 thousand products, 70 thousand users and 1.6 million view events.

pio build is done successfully but in pio train it goes unresponsive at stage 13 since I have tried 3 times at waited for 6 to 7 hours but it stuck at the stage 13 below is the logs

[INFO] [Console$] Using existing engine manifest JSON at /home/dau/PredictionIO/similar_product/manifest.json
[INFO] [Runner$] Submission command: /home/dau/PredictionIO/vendors/spark-1.6.0/bin/spark-submit --class io.prediction.workflow.CreateWorkflow --jars file:/home/dau/PredictionIO/similar_product/target/scala-2.10/template-scala-parallel-similarproduct-assembly-0.1-SNAPSHOT-deps.jar,file:/home/dau/PredictionIO/similar_product/target/scala-2.10/template-scala-parallel-similarproduct_2.10-0.1-SNAPSHOT.jar --files file:/home/dau/PredictionIO/conf/log4j.properties,file:/home/dau/PredictionIO/vendors/hbase-1.1.2/conf/hbase-site.xml --driver-class-path /home/dau/PredictionIO/conf:/home/dau/PredictionIO/lib/postgresql-9.4-1204.jdbc41.jar:/home/dau/PredictionIO/lib/mysql-connector-java-5.1.37.jar:/home/dau/PredictionIO/vendors/hbase-1.1.2/conf file:/home/dau/PredictionIO/lib/pio-assembly-0.9.6.jar --engine-id TvPK9sthW8Hjh6lyz0PRqtCb6IIzOLSF --engine-version 9c974b7a0cc9a0a9c97387bbacb6486db3ebb738 --engine-variant file:/home/dau/PredictionIO/similar_product/engine.json --verbosity 0 --json-extractor Both --env PIO_STORAGE_SOURCES_HBASE_TYPE=hbase,PIO_ENV_LOADED=1,PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta,PIO_FS_BASEDIR=/home/dau/.pio_store,PIO_STORAGE_SOURCES_HBASE_HOME=/home/dau/PredictionIO/vendors/hbase-1.1.2,PIO_HOME=/home/dau/PredictionIO,PIO_FS_ENGINESDIR=/home/dau/.pio_store/engines,PIO_STORAGE_SOURCES_LOCALFS_PATH=/home/dau/.pio_store/models,PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch,PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH,PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS,PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event,PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=/home/dau/PredictionIO/vendors/elasticsearch-1.7.3,PIO_FS_TMPDIR=/home/dau/.pio_store/tmp,PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model,PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE,PIO_CONF_DIR=/home/dau/PredictionIO/conf,PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs
[INFO] [Engine] Extracting datasource params...
[INFO] [WorkflowUtils$] No 'name' is found. Default empty String will be used.
[INFO] [Engine] Datasource params: (,DataSourceParams(rw_store))
[INFO] [Engine] Extracting preparator params...
[INFO] [Engine] Preparator params: (,Empty)
[INFO] [Engine] Extracting serving params...
[INFO] [Engine] Serving params: (,Empty)
[WARN] [Utils] Your hostname, dau-lap resolves to a loopback address: 127.0.1.1; using 192.168.43.91 instead (on interface wlan0)
[WARN] [Utils] Set SPARK_LOCAL_IP if you need to bind to another address
[INFO] [Remoting] Starting remoting
[INFO] [Remoting] Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@192.168.43.91:40349]
[INFO] [Engine$] EngineWorkflow.train
[INFO] [Engine$] DataSource: com.company.DataSource@7cfb8e98
[INFO] [Engine$] Preparator: com.company.Preparator@1ec88aa1
[INFO] [Engine$] AlgorithmList: List(com.company.ALSAlgorithm@3030836d)
[INFO] [Engine$] Data sanity check is on.
[WARN] [TableInputFormatBase] Cannot resolve the host name for dau-lap/127.0.1.1 because of javax.naming.NameNotFoundException: DNS name not found [response code 3]; remaining name '1.1.0.127.in-addr.arpa'
[WARN] [TableInputFormatBase] Cannot resolve the host name for dau-lap/127.0.1.1 because of javax.naming.NameNotFoundException: DNS name not found [response code 3]; remaining name '1.1.0.127.in-addr.arpa'
[WARN] [TableInputFormatBase] Cannot resolve the host name for dau-lap/127.0.1.1 because of javax.naming.NameNotFoundException: DNS name not found [response code 3]; remaining name '1.1.0.127.in-addr.arpa'
[WARN] [TableInputFormatBase] Cannot resolve the host name for dau-lap/127.0.1.1 because of javax.naming.NameNotFoundException: DNS name not found [response code 3]; remaining name '1.1.0.127.in-addr.arpa'
[INFO] [Engine$] com.company.TrainingData does not support data sanity check. Skipping check.
[INFO] [Engine$] com.company.PreparedData does not support data sanity check. Skipping check.
[WARN] [TableInputFormatBase] Cannot resolve the host name for dau-lap/127.0.1.1 because of javax.naming.NameNotFoundException: DNS name not found [response code 3]; remaining name '1.1.0.127.in-addr.arpa'
[WARN] [TableInputFormatBase] Cannot resolve the host name for dau-lap/127.0.1.1 because of javax.naming.NameNotFoundException: DNS name not found [response code 3]; remaining name '1.1.0.127.in-addr.arpa'
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 513 to Int index.    
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 513 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 0 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 513 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 513 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 513 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 513 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 513 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 513 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 513 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 513 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 513 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 513 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 513 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 513 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 513 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 513 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 513 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 513 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 513 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 513 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 513 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 513 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 513 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 513 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 513 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 3055 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 3055 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 3055 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 513 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 513 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 513 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 513 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 222 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 222 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 416 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 0 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 513 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 513 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 513 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 513 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 513 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 513 to Int index.
[Stage 13:>                                                         (0 + 1) / 2][INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 0 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 484 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 513 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 513 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 222 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 414 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 513 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 513 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 513 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 513 to Int index.
[INFO] [ALSAlgorithm] Couldn't convert nonexistent item ID 513 to Int index.
[Stage 13:=============================>                            (1 + 0) / 2]

Here is engine.json

{
  "id": "default",
  "description": "Default settings",
  "engineFactory": "com.company.SimilarProductEngine",
  "datasource": {
    "params" : {
      "appName": "store"
    }
  },
  "algorithms": [
    {
      "name": "als",
      "params": {
        "rank": 10,
        "numIterations" : 20,
        "lambda": 0.01,
        "seed": 3
      }
    }
  ]
}
Dau
  • 8,578
  • 4
  • 23
  • 48
  • You've not shown your code/command line. You've lots of warnings - are those non-existent line items an issue? Does it work on a smaller dataset? Etc, etc. Basically, you're not giving us much to work with here... – The Archetypal Paul Jun 11 '16 at 14:36
  • @TheArchetypalPaul I have updated engine.json here. I have used simple commands like pio build and pio train. Complementary purchase template worked successfully on the same dataset. Please let me know what other info you want – Dau Jun 11 '16 at 16:29
  • And what about a small data set for this template? – The Archetypal Paul Jun 11 '16 at 16:29
  • @TheArchetypalPaul I will try it too then inform you – Dau Jun 11 '16 at 16:32
  • 1
    Looks like other people have had this problem; https://groups.google.com/forum/#!topic/predictionio-user/a00QVzPIvuM and https://groups.google.com/forum/#!topic/predictionio-user/sTgc_AnClXo The second link suggests setting up a standalone Spark cluster helps a lot.. – The Archetypal Paul Jun 11 '16 at 16:34

1 Answers1

0

you should define three types of events.user, item and view event.i think you define view event on an specific items that does not have their corresponding item.

htb
  • 1
  • 1