3

I've been using a sqoop-import command like this:

sqoop import --connect jdbc:oracle:thin:@${machine}:${port}/${schema} --username ${user} --password ${pw} --table "${table}" --columns "${cols}" --where "${machine}" --m 1 --fields-terminated-by ';' --target-dir ${dir} --hive-table "${hive_table}"

It has worked properly until now, when I receive this message:

Picked up _JAVA_OPTIONS: -Djava.io.tmpdir=/data/tmp
Java HotSpot(TM) 64-Bit Server VM warning: Insufficient space for shared memory file:
   64215
Try using the -Djava.io.tmpdir= option to select an alternate temp location.

Indeed my /tmp is full:

/dev/mapper/system-lv_tmp 5136000 4875796 0 100% /tmp

I cannot delete files that are not mine in /tmp so i can't free up enough space there.

I've tried changing the _JAVA_OPTIONS to another directory but still get the same error.

Is there any way to make the command use a different directory for temporal files?

franklinsijo
  • 17,784
  • 4
  • 45
  • 63
David
  • 33
  • 4
  • Is `/data/tmp` the same as `/tmp`? If you still get the same error (`Picked up _JAVA_OPTIONS: -Djava.io.tmpdir=/data/tmp`) even if you've supplied a different directory in `_JAVA_OPTIONS` it seems the `_JAVA_OPTIONS` you supply and the options `sqoop` uses aren't the same. – Ted Lyngmo Apr 01 '20 at 09:14
  • Indeed /data/tmp and /tmp are different directories... The error message only appears when /tmp is full. /data/tmp has space available... The thing is, can i tell sqoop to use another directory that is not /tmp in any way? – David Apr 01 '20 at 09:33
  • I thought I found a way by looking in the source code, but apparently it didn't work so I removed the answer. – Ted Lyngmo Apr 01 '20 at 14:07
  • One idea if everything else fails is to just download the source, change all lines containing `/tmp` to `/data/tmp` and then compile your own version of `sqoop`. – Ted Lyngmo Apr 01 '20 at 14:17

2 Answers2

0

This is caused by the Java property java.io.tmpdir and has nothing to do with Sqoop.

From the Java Docs:

The default temporary-file directory is specified by the system property java.io.tmpdir.

Sqoop is not setting the value for tmpdir. But you can override this value when running your sqoop import command. 

sqoop import -Djava.io.tmpdir=/path/to/new/tmp ...
franklinsijo
  • 17,784
  • 4
  • 45
  • 63
  • I already changed the _JAVA_OPTIONS var to another directory as it was shown by the command when launched, but still it created files on /tmp. /data/tmp is not the same as /tmp – David Apr 01 '20 at 15:20
  • Sure, this is launched from bash, so in that environment i set _JAVA_OPTIONS="-Djava.io.tmpdir=/path/to/new/tmp" – David Apr 01 '20 at 15:23
  • The sqoop -import would show when launch: Picked up _JAVA_OPTIONS: -Djava.io.tmpdir=/path/to/new/tmp. I believe this to be the same that you answered. – David Apr 01 '20 at 15:24
  • But do you get the same error even after it says that it has picked up a new value for `java.io.tmpdir`? – franklinsijo Apr 01 '20 at 15:29
  • What is the OS and java version? – franklinsijo Apr 01 '20 at 15:31
  • Yes, i get the same error no matter what is the value of `java.io.tmpdir` – David Apr 01 '20 at 15:32
  • 1
    OS is Linux and Java version is 1.8.0_60 – David Apr 01 '20 at 15:34
  • Looks like the JVM is ignoring the value set by the program or the environment. Let us try this `export TMPDIR=/path/to/new/tmp`. Also, did you try running as mentioned in the answer, did it behave the same ? – franklinsijo Apr 01 '20 at 16:34
  • Neither of them work. The behaviour is the same, keeps going to /tmp – David Apr 02 '20 at 15:34
0

Maybe your problem is the code generated for Sqoop, there are several code generation control arguments like:

enter image description here

For example, you could try this:

sqoop import \
--connect jdbc:oracle:thin:@${machine}:${port}/${schema} \
--username ${user} \
--password ${pw} \
--table "${table}" \
--columns "${cols}" \
--where "${machine}" \
--m 1 \
--fields-terminated-by ';' \
--target-dir ${dir} \
--hive-table "${hive_table}" \
--outdir /home/user/outdir_directory \
--bindir /home/user/bindir_directory

and the java files generated for Sqoop are going to go to {outdir_directory and bindir_directory} so you can delete these whenever you want.

Chema
  • 2,748
  • 2
  • 13
  • 24
  • have you created /home/user/outdir_directory and --bindir /home/user/bindir_directory directories? because if them don't exists sqoop doesn't create them for you. – Chema Apr 03 '20 at 07:34
  • and you have to be sure that sqoop has the right permissions to write in there – Chema Apr 03 '20 at 12:00