Reading and writing Large files in C

Question

I wanted to know the issue in the following code. This is used to first open a file of size 800Mb to fill in the name variable and later used to access the data stored in it by reading indices from another file. The problem is that after reading and filling the array name, on accessing any element from it gives a seg fault. When the same code was tested on smaller data size, it worked. What could be the reason? The hardware on which I am running this is a 4gb RAM,32 bit linux version on intel i5 chip.

#include <stdio.h>
#define MAXINT 61578414
int main(int argc,char** argv){
    printf("Starting \n");    

    FILE* fp1 = fopen(argv[1],"r");
    FILE* fp2 = fopen(argv[2],"r");


    char** name;
    name = (char**)malloc(MAXINT*sizeof(char*));

    char* tname;

    int i = 0;
    int tmp1;


    //reading to fill in name
    while(i < MAXINT){
        name[i] = (char*)malloc(20);
        fscanf(fp1,"%d%s",&tmp1,name[i]);
        i++;
    }

    //accessing elements from  name
    int i1,i2;
    while(!feof(fp2)){
        fscanf(fp2,"%d%d",&i1,&i2);
        fprintf(stdout,"%s %s\n",*(name+i1),*(name+i2));
    }

}

1) include 2) dont cast malloc()s return value. 3) feof() is always wrong. 4) `name[i1]` is easier to read than `*(name+i1)` 5) (minor) main() should return an int value, in most cases zero. — wildplasser, Apr 13 '13 at 13:19
also I don't see the allocated memory being free'd anywhere? — Pankrates, Apr 13 '13 at 13:24
Yes , will incorporate the changes but issue is that at `name[i1]` I am getting a segmentation fault. — ImWatchingYou, Apr 13 '13 at 13:27
@MartinWickman The above condition is not needed.. `i1` and `i2` should be within `(0, MAXINT - 1)` — ImWatchingYou, Apr 13 '13 at 13:57

score 1 · Answer 1 · answered Apr 13 '13 at 13:20

1

There is one potential problem with your allocation of name.

char** name;
name = (char**)malloc(MAXINT*sizeof(char*));

name is defined to be a pointer to pointer, which is an array of pointers. The code is allocating MAXINT number of pointers, which is a very large number. You need to check if next was successfully allocated or a NULL pointer was removed i.e. whether malloc was successful or not.

Similarly, name[i] = malloc(20) should also be checked for NULL pointer as your system may potentially run out of memory.

answered Apr 13 '13 at 13:20

Ganesh

5,880
2
36
54

Yes , will incorporate the changes but when I checked it is able to allocate that much space at call of every `malloc` – ImWatchingYou Apr 13 '13 at 13:28
@ImWatchingYou If you wish to print the string, can't this statement `fprintf(stdout,"%s %s\n",*(name+i1),*(name+i2))` be changed to `fprintf(stdout,"%s %s\n", name[i1],name[i2])`. Any reason why you opted for your code? – Ganesh Apr 13 '13 at 13:33
When I tried `name[i1]` is was throwing a segmentation fault. I taught that `*(name+ i1 )` may help in avoiding it.But later realized that both were identical – ImWatchingYou Apr 13 '13 at 13:44
@ImWatchingYou.. Have you checked the value of `i1` and `i2`? Can you confirm that this is well within the range of `(0, MAXINT - 1)` in all cases? Also, can you paste a sample of your input and index files to understand the problem a bit more? – Ganesh Apr 13 '13 at 13:51
"MAXINT number of pointers, which is a very large number" -- MAXINT != MAX_INT; the former is defined in the program and it's not all that large. – Jim Balter Apr 13 '13 at 14:41
@JimBalter..Assuming a `32-bit` system, the total allocation is `MAXINT * 4` for pointers + `MAXINT * 20` for each of the strings which is a total of `1.34 GiB` allocation, where `MAXINT = 61578414` – Ganesh Apr 13 '13 at 14:57

score 1 · Answer 2 · edited May 23 '17 at 11:43

Is this on Linux? In that case, (by default) non-NULL values from malloc doesn't tell you if you actually have access to the amount of memory you request. In practice, this means you can malloc far more memory than you have available. It's only when you actually access it, that the memory it going to be allocated.

So write a simple loop which just reads or writes to each byte you malloc'ed and see if that crashes. If it does, than well that's your problem.

As a side-note, have you considered using mmap() instead to deal with large file/memory allocation?

score 0 · Answer 3 · answered Apr 13 '13 at 14:22

0

I imagine that name = (char**)malloc(MAXINT*sizeof(char*)); actually fails, you should check for a NULL return value there.

answered Apr 13 '13 at 14:22

rmn

2,386
1
14
21

Reading and writing Large files in C

3 Answers3