6

I created a pipeline that strings multiple programs together, unfortunately these programs are creating a huge amount of temporary files in the /tmp folder and when using large datasets my pipeline crashes because the /tmp folder fills up.

How do I export temporary files so that they are created in my current working directory where the pipeline is being run and not in the /tmp folder?

Currently I have tried to export the TMPDIR env variable to an already created directory /work in my current working directory, but the temporary files are still being created in the /tmp folder:

export TMPDIR=$(mktemp -d --tmpdir=/work)
<script>
rm -rf $TMPDIR

The programs do not have the option to set different output folders for temporary files created.

Christopher Bottoms
  • 11,218
  • 8
  • 50
  • 99
System
  • 295
  • 1
  • 4
  • 13
  • What error message do you get? Please always mention error messages when they are available. – Christopher Bottoms Sep 15 '16 at 20:52
  • The error message for the pipeline? It's a typical IOError: No space available on device error that's very common when the /tmp file is full. – System Sep 15 '16 at 21:02
  • How much space is left on your hard drive (i.e. `df -h . ` )? – Christopher Bottoms Sep 15 '16 at 21:04
  • I have two 4TB hard-drives where I do most of the scratch work. One is completely empty and the other is 50% full. My /tmp directory only has 9gb of free space which is where the problem stems from. – System Sep 15 '16 at 21:05

1 Answers1

4

Just change /work to work if the directory work is in your current directory. /work means that you have a top-level directory named /work. Without the forward slash, it will be a relative directory.

I just tested this code on my computer. No files were written to /tmp that I noticed:

mkdir work
export TMPDIR=$(mktemp -d --tmpdir=work)
ls work
# tmp.AWA4dTERha
rm -rf $TMPDIR
ls work
# --no output--
Christopher Bottoms
  • 11,218
  • 8
  • 50
  • 99