1

I am running Jupyter notebook on a system with 64gb RAM, 32 cores and 500GB disk space.

Around 700k documents are to be modeled into 600 topics. The vocabulary size is 48000 words. 100 iterations were used.

spark = SparkSession.builder.appName('LDA').master("local[*]").config("spark.local.dir", "/data/Data/allYears/tempAll").config("spark.driver.memory","50g").config("spark.executor.memory","50g").getOrCreate()

dataset = spark.read.format("libsvm").load("libsm_file.txt")

lda = LDA(k=600, maxIter=100 ,  optimizer='em' , seed=2 )

lda.setDocConcentration([1.01])
lda.setTopicConcentration(1.001)
model = lda.fit(dataset)

Disk quota exceeded error comes after 10 hours of run

James Z
  • 12,209
  • 10
  • 24
  • 44
adihere
  • 81
  • 9
  • 1
    Please don't use Indian measurements here. Other people will just be confused about what are "lakh documents". – James Z Apr 27 '19 at 17:10

1 Answers1

1

You mentioned that the error message you encountered indicated that the disk quota has been exceeded. I suspect that Spark is shuffling data to disk and that disk is out of space.

To mitigate this, you should try explicitly passing --conf spark.local.dir=<path to disk with space> to a location with sufficient space. This parameter specifies what path Spark will use to write temporary data to disk (e.g. when writing shuffle data between stages of your job). Even if your input and output data are not particularly large, certain algorithms can generate a very large amount of shuffle data.

You could also consider monitoring the allocated/free space of this path using du while running your job to get more information on how much intermediate data is being written. This would confirm that a high amount of shuffle data exhausting the available disk space is the issue.

dchristle
  • 132
  • 1
  • 6
  • I had tried running spark with path pointing to free disk (of space around 500 GB) I am not sure why its generating so much of temporary files to exceed the given space , since the vocabulary size is around 48000 words alone and topics to be found is 600 – adihere Apr 28 '19 at 19:26
  • Try monitoring the disk space as your job executes -- that way, you can see how much disk space is being taken up by the shuffle data. For what it's worth, I recently ran a job that started with a ~200GB input, created a ~2 GB output, and generated 78 TB worth of shuffle data. It's algorithm-dependent, so monitoring your disk space while it's executing will confirm that this is the issue. – dchristle Apr 29 '19 at 00:11
  • @adihere Did you eventually find the solution to your issue? – dchristle May 26 '19 at 16:54