2

My Java Application returns following Exception when saving a new file in /opt/wso2 on a CentOS 6.4:

Caused by java.io.FileNotFoundException: ... (No space left on device)

Caused by: java.io.FileNotFoundException: /opt/wso2/FrameworkFiles/trk_2014062500042488825_TRCK_PatfallHospis_pFromHospis_66601fb3-a03c-4149-93c3-6892e0a10fea.txt (No space left on device)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:212)
at java.io.FileOutputStream.<init>(FileOutputStream.java:99)
at com.avintis.esb.framework.adapter.wso2.FrameworkAdapterWSO2.sendMessages(FrameworkAdapterWSO2.java:634)
... 23 more

But when I run df -a I can see that the partition still has plenty of space available:

[root@stzsi466 wso2]# df -a
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/vg_stzsi466-lv_root
                      12054824   2116092   9326380  19% /
proc                         0         0         0   -  /proc
sysfs                        0         0         0   -  /sys
devpts                       0         0         0   -  /dev/pts
tmpfs                  4030764         0   4030764   0% /dev/shm
/dev/sda1               495844     53858    416386  12% /boot
/dev/sdb1             51605436   1424288  47559744   3% /opt/wso2
none                         0         0         0   -  /proc/sys/fs/binfmt_misc

[root@stzsi466 ~]# df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/vg_stzsi466-lv_root
                      765536   45181  720355    6% /
tmpfs                1007691       1 1007690    1% /dev/shm
/dev/sda1             128016      44  127972    1% /boot
/dev/sdb1            3276800    6137 3270663    1% /opt/wso2

What is the problem here? Is it caused by the Java on CentOS 6.4? I have another server running Redhat REHL 6.4 and all works fine - same Java etc.

Does anyone know of this problem?

FiveO
  • 133
  • 1
  • 7
  • Is this a standalone box or is it a shared cloud 'instance'? I have seen virtual servers do this. – Jeff Clayton Aug 19 '14 at 12:43
  • 1
    Please, clirify your question in following: - FS type in use - mount params - file permissions of target file directory and it's parents. This will help to narrow correct answer. Also: check inodes with command: df -i – cgi Aug 19 '14 at 12:41
  • In the virtual server that was problematic, the server instance was allocated a specific amount of hard drive space. The server that contained it also covered automatic backups. The containing server took up more space and encroached on the virtual server's space. The virtual server reported what it was told it should have, but could not expand to it's full size. VMWare based servers are susceptible for example, and the cause could also have been a swap space or other server instance 'crowding it out'. The virtual server I am referring to was a CentOS 6.5 instance as well. – Jeff Clayton Aug 19 '14 at 12:54
  • In a VM it is often pre-set to tell the internal server that it has a certain space, but to grow to that size as files are added, instead of requisitioning the whole block of space so it doesn't take up space on the containing box until needed. If your server is standalone, this should not happen of course. – Jeff Clayton Aug 19 '14 at 12:56
  • Thanks for the responses. YES it is a Virtual Server. I'll post now the different FS informations... – FiveO Aug 19 '14 at 13:15
  • FS is ext4, Permission is 775 (parent too) and the user is the owner. – FiveO Aug 19 '14 at 13:24
  • What's the memory utilization at the time this issue happens (`free -m`)? – gtirloni Aug 19 '14 at 13:30
  • What does `quota -v` return? Perhaps your user account has enforced file system quotas. – Janne Pikkarainen Aug 19 '14 at 13:41
  • memory utilization is not really high, more than 50% available. quoata comand not found. – FiveO Aug 19 '14 at 13:45
  • Can you 'touch' a file on that device? If it's a virtual disk, is there space on the relevant datastore? – Sobrique Aug 19 '14 at 13:50
  • Please copy the text into the question. A picture of the text is not very useful. Also, what percentage of inodes are in use? – kasperd Aug 19 '14 at 14:22

1 Answers1

0

So from the stack trace we can see that when the FileOutputStream object is being created, inside that code, it makes a call to open() that fails and then it throws a FileNotFoundException with the "No space left on device" error message.

It's interesting to look at the code that is failing (FileOutputStream.java):

       public FileOutputStream(File file, boolean append)
           throws FileNotFoundException
       {
           String name = (file != null ? file.getPath() : null);
           SecurityManager security = System.getSecurityManager();
           if (security != null) {
               security.checkWrite(name);
           }
           if (name == null) {
               throw new NullPointerException();
           }
           this.fd = new FileDescriptor();
           this.append = append;

           fd.incrementAndGetUseCount();
           open(name, append);                /* HERE */
       }

So this is a call to native code (FileOutputStream_md.c):

Java_java_io_FileOutputStream_open(JNIEnv *env, jobject this,
                                   jstring path, jboolean append) {
    fileOpen(env, this, path, fos_fd,
             O_WRONLY | O_CREAT | (append ? O_APPEND : O_TRUNC));
}

And fileOpen (io_util_md.c):

fileOpen(JNIEnv *env, jobject this, jstring path, jfieldID fid, int flags)
{
    WITH_PLATFORM_STRING(env, path, ps) {
        FD fd;

#ifdef __linux__
        /* Remove trailing slashes, since the kernel won't */
        char *p = (char *)ps + strlen(ps) - 1;
        while ((p > ps) && (*p == '/'))
            *p-- = '\0';
#endif
        fd = JVM_Open(ps, flags, 0666);              /* HERE IS WHERE THINGS BREAK */
        if (fd >= 0) {
            SET_FD(this, fd, fid);
        } else {
            throwFileNotFoundException(env, path);   /* EXCEPTION */
        }
    } END_PLATFORM_STRING(env, ps);
}

And JVM_Open (jvm.cpp):

JVM_LEAF(jint, JVM_Open(const char *fname, jint flags, jint mode))
  JVMWrapper2("JVM_Open (%s)", fname);

  //%note jvm_r6
  int result = os::open(fname, flags, mode);
  if (result >= 0) {
    return result;
  } else {
    switch(errno) {
      case EEXIST:
        return JVM_EEXIST;       /* returns -100, will trigger Exception */
      default:
        return -1;               /* returns -1, will trigger Exception */
    }
  }
JVM_END

And os::open is just a wrapper around fdopen(2) syscall.

Can you reproduce this error whenever you want? If so, please run the command with strace -e trace=fopen,fdopen,freopen and post results (or attach to running process with strace -p <PID>).

gtirloni
  • 5,746
  • 3
  • 25
  • 52