Efficient way of writing large variable number of integers from array to text file

Question

I have a program that results in an integer array of variable size and I need to write the elements to a text file. The elements need to be on the same line in the file. This is a minimal example of what I am currently doing. I'm using the approach in this post https://stackoverflow.com/a/30234430/10163981

FILE *file = fopen("file.txt","w");
int nLines = 1000;
char breakstr[]="\n";

for(; ix<N;  ix++){
    char s[nLines*13];
    for(int jx = 0 ; jx<nLines; jx++){
        index += sprintf(&s[index],"%03i %03i %03i ", array[ix][jx],  array[ix][jx], array[ix][jx]); 
    // I need jx:th element in repeats of three, and may need to modify it externally for printing
    }
    fwrite(s,sizeof(char),strlen(s), file);
    fwrite(breakstr,sizeof(char),strlen(breakstr), file);
}
fclose(file);

I am formatting the array contents as a string and using fwrite, as this method has been given to me as a requirement. My problem is that this implementation that I am using is way too slow. I have also tried using shorter strings and writing for each iteration, but this even slower. There is not much I can do with regards to the outer ix loop, as the initial value of ix is variable. I included it for the same of completeness.

nLines is expected to reach as high as 10000 at most.

It shouldn't really matter much how you do the loops. `stdio` uses buffered output, and the bottleneck is when it flushes the buffer. — Barmar, Oct 20 '20 at 22:41
You could try formatting each array element just once, then concatenate that to `s` 3 times. number formatting is a little slower than string concatenation. — Barmar, Oct 20 '20 at 22:50
So you're formatting 100,000,000 numbers 3 times apiece, and writing over a gigabyte. How fast do you expect that to be? — Barmar, Oct 20 '20 at 22:51
My 2 cents: `strlen(breakstr)` -> `1`. But it might not matter. — Jabberwocky, Oct 20 '20 at 22:52
@Jabberwocky I'll bet the compiler already does that optimization, since it can see that `breakstr` is never modified. — Barmar, Oct 20 '20 at 22:53
The time case I have is for 1000*1000=1e6 which I need to be faster than 0.04s — dan, Oct 20 '20 at 22:54
IMHO the easiest way of optimizing by hand here is getting rid of the sprintf. — Jabberwocky, Oct 20 '20 at 22:55
See https://stackoverflow.com/questions/11819837/converting-integer-to-string-in-c-without-sprintf for a `sprintf()` replacement. — Barmar, Oct 20 '20 at 22:56
i would focus on the read in process, because thats for sure slower. — Ol Sen, Oct 20 '20 at 22:58
maybe try to declare the s array outside of the for, and also the jx variable outside of the for. so no need to be istantiated at every loop — Andrew Nic, Oct 20 '20 at 23:40
`char s[nLines*13]` is computed everytime, and its always the same size so you an create a variable outside and set `variable = nLines*13;` and use `s[variable]` — Andrew Nic, Oct 20 '20 at 23:42

Efficient way of writing large variable number of integers from array to text file

0 Answers0