0

I have spark yarn client submitting jobs and when it does that, it creates a directory under my "spark.local.dir" which has files like:

__spark_conf__8681611713144350374.zip
__spark_libs__4985837356751625488.zip

Is there a way these can be automatically cleaned? Whenever I submit a spark job I see new entries for these again in the same folder. This is flooding up my directory what should I set to make this clear automatically?

I have looked at a couple of links online even on SO but couldn't find a solution to this problem. All I found was a way to specify the dir path by "spark.local.dir".

Harinder
  • 11,776
  • 16
  • 70
  • 126

1 Answers1

0

Three SPARK_WORKER_OPTS exists to support the worker application folder cleanup, copied here for further reference: from Spark Doc

  1. spark.worker.cleanup.enabled, default value is false, Enable periodic cleanup of worker / application directories. Note that this only affects standalone mode, as YARN works differently. Only the directories of stopped applications are cleaned up.

  2. spark.worker.cleanup.interval, default is 1800, i.e. 30 minutes, Controls the interval, in seconds, at which the worker cleans up old application work dirs on the local machine.

  3. spark.worker.cleanup.appDataTtl, default is 7*24*3600 (7 days), The number of seconds to retain application work directories on each worker. This is a Time To Live and should depend on the amount of available disk space you have. Application logs and jars are downloaded to each application work dir. Over time, the work dirs can quickly fill up disk space, especially if you run jobs very frequently.

Ajay Kharade
  • 1,469
  • 1
  • 17
  • 31