0

When using spark-submit, I need to send the dependencies with --packages. Which package should I use?

I tried making an uber jar which contain the dependencies, but I get the following error:

java.lang.IllegalArgumentException: Please specify an existing file

The error is the result of the following code:

String path = "hdfs:///user/data.txt";
SentenceIterator iter = new LineSentenceIterator(new File(path));
blpasd
  • 409
  • 7
  • 18

1 Answers1

2

Deeplearning4j actually has its own spark version of word2vec. Check in our examples here: https://github.com/deeplearning4j/deeplearning4j/blob/master/deeplearning4j-scaleout/spark/dl4j-spark-nlp/src/test/java/org/deeplearning4j/spark/models/embeddings/word2vec/Word2VecTest.java#L57

Adam Gibson
  • 3,055
  • 1
  • 10
  • 12
  • The link is dead. Try this instead: https://github.com/deeplearning4j/dl4j-examples/blob/57befb80aeb53523413c363a4edb15d682d59836/dl4j-examples/src/main/java/org/deeplearning4j/examples/nlp/word2vec/Word2VecRawTextExample.java – Max Murphy Jun 08 '18 at 06:21