8

I am reading about 6000 text-files into memory with the following code in a loop:

void readDocs(const char *dir, char **array){
DIR *dp = opendir(dir);;
struct dirent *ep;
struct stat st;
static uint count = 0;
if (dp != NULL){
  while (ep = readdir(dp)){    // crawl through directory
    char name[strlen(dir) + strlen(ep->d_name) + 2];
    sprintf(name, "%s/%s", dir, ep->d_name);

    if(ep->d_type == DT_REG){  // regular file
      stat(name, &st);
      array[count] = (char*) malloc(st.st_size);
      int f;
      if((f = open(name, O_RDONLY)) < 0) perror("open:  ");
      read(f, array[count], st.st_size));
      if(close(f) < 0) perror("close: ");
      ++count;
    }    

    else if(ep->d_type == DT_DIR && strcmp(ep->d_name, "..") && strcmp(ep->d_name, "."))
      // go recursive through sub directories
      readDocs(name, array);
  }
}
}

In iteration 2826 i get an "Too many open files" error when opening the 2826th file. No error occured in the close operation until this point.
Since it always hangs in the 2826th iteration i do not believe that i should wait until a file is realy closed after calling close();
I had the same issue using fopen, fread and fclose.
I don't think it has to do with the context of this snippet but if you do i will provide it.
Thanks for your time!

EDIT:
I put the program to sleep and checked /proc//fd/ (thanks to nos). Like you suspected there were exactly 1024 file descriptors which i found to be a usual limit.
+ i gave you the whole function which reads documents out of a directory and all subdirectories
+ the program runs on Linux! Sorry for forgetting that!

Patrick B.
  • 11,773
  • 8
  • 58
  • 101
Callahan
  • 474
  • 1
  • 9
  • 22
  • 6
    Somewhere, you havn't close()'d the file, maybe on an error path. If this is on linux, take a look in /proc//fd/ , you will atleast see which filedescriptors you've failed to close() – nos Aug 24 '11 at 12:56
  • as if the system was actually closing the files after the call to close(). Under which system are you running this program? – Shlublu Aug 24 '11 at 12:58
  • 1
    We reaaaaaally need more code and details here. – Raveline Aug 24 '11 at 13:11
  • @nos i found out that there are indeed several files left open. question is why. – Callahan Aug 24 '11 at 13:46

4 Answers4

12

You need to call closedir() after having looped. Opening a directory also consumes a file-descriptor.

Patrick B.
  • 11,773
  • 8
  • 58
  • 101
  • This is it. This will make recursing a bit complicated but Thank you all! – Callahan Aug 24 '11 at 14:17
  • 2
    How come? Just add closedir(dp) after the while-loop has ended. Or do you have 1000-levels of directories? You could also make a list of all files recursively and then open them one by one afterwards outside the recursion. – Patrick B. Aug 24 '11 at 14:19
2

You may be hitting the OS limit for # of open files allowed. Not knowing which OS you are using, you should google your OS + "too many open files" to find out how to fix this. Here is one result for linux, http://lj4newbies.blogspot.com/2007/04/too-many-open-files.html

Matt Arnett
  • 21
  • 3
  • 1
    Increasing the limit foropen files on the OS is the wrong way to go. He should rather check why he is not closing his files. – RedX Aug 24 '11 at 13:30
1

I solved the problem by adding to /etc/security/limits.conf

* soft nofile 40960

* hard nofile 102400

Problem was when login to debian it shows ulimit -n 40960, but when su user, it's again 1024. Need uncomment one row on /etc/pam.d/su

session required pam_limits.so

Then always needed limits

Anuj Balan
  • 7,629
  • 23
  • 58
  • 92
witeX
  • 55
  • 7
0

You should call closedir() as it opendir also returns descriptor as in linux system maximum this much number of time /proc/sys/fs/file-max file can be opened although you can increase/decrease this number

Anshul garg
  • 233
  • 2
  • 6