Iterate C char array of size 100000 takes huge time

Question

Just try to iterate a character array of size 100000 but it takes 8-9 sec. But for int its takes some milli second.

    int i;
    char str[100005];
    str[0] = '\0';
    for(i=1;i<10000;i++){
        strcat(str, "HelloWorld");
    }
    for(i=1;i<strlen(str);i++){
        str[i]=str[i-1];
    }

Undefined behavior for using the value of an object with automatic storage duration while it is indeterminate. — EOF, Apr 10 '20 at 20:38
Yes. If finds the end of the current string and copies in the contents of the new one. — Mad Physicist, Apr 10 '20 at 20:38
@MadPhysicist used `str[0] = '\0';` but still it takes huge time — Eklavya, Apr 10 '20 at 20:41
@AbinashGhosh. That's for correctness. There is a good chance that your allocated memory is already zero-filled. The issue is with the functions you are using. — Mad Physicist, Apr 10 '20 at 20:44
Don't forget to select an answer and remove your question from the unanswered queue. — Mad Physicist, Apr 11 '20 at 02:32

score 4 · Accepted Answer · answered Apr 10 '20 at 20:40

Let's look at this code:

for(i = 1; i < 10000; i++){
    strcat(str, "HelloWorld");
}

Think about how that call to strcat works. The strcat function works by scanning forward in the string until it finds the null terminator character, then writing the second argument there. That's pretty fast if your string is fairly short - say, it's about 10-20 characters long - but if the string you're appending onto is long (say, 10,000 characters), then strcat will spend a lot of time simply scanning to the end of the string to find the place at which to append the string. This means that each call to strcat in this function will take longer than the previous one, and eventually that last call is fairly slow.

So how can you speed this up? One option would be to store a pointer to the null terminator in the str array so that you can tell strcat where to start looking. Here's what that might look like:

size_t length = strlen("HelloWorld");
char* nextSpot = str;
for(i = 1; i < 10000; i++){
    strcat(nextSpot, "HelloWorld");
    nextSpot += length;
}

This means that the strcat call will always pick up right where the last one left off, speeding things up.

Hope this helps!

Actually I have to read from a file a do some operation for every char, how to speed up then ? — Eklavya, Apr 11 '20 at 04:29

Mad Physicist · Answer 2 · 2020-04-10T21:00:55.083

There are some problems with your code. The first is that you are calling strcat on a destination array that has not been initialized. You need to set str[0] = '\0'; first.

The other problems are algorithmic. To know where to insert the target string, strcat has to do strlen or equivalent every time you call it. That means that it is stepping through O(n) characters every time. That makes your algorithm ~O(n^2). Then you go and call strlen again on every single character, with the same penalty. And keep in mind that the second loop is just setting the entire array to 'H'.

If you want to copy a string 10000 times, you are better off pre-computing its length and just stepping along the buffer:

char str[100005];
const char *template = "HelloWorld";
int n = strlen(template);
char *p = str;
for(int i = 0; i < 10000; i++) {
    strncpy(p, template, n);
    p += n;
}
*p = '\0';

I am using strncpy here because it avoids copying the terminating '\0' of template for every copy you make.

An alternative approach would be to use modulo division (which would probably be slower) to place the characters:

char str[100005];
const char *template = "HelloWorld";
int n = strlen(template);
int end = n * 10000;
for(int i = i; i < end; i++) {
    str[i] = template[i % n];
}

Both approaches take a couple of orders of magnitude less time on my machine than your original loop, because they step through the array exactly once.

The final loop copies the previous character over and over into the remainder of the buffer. Since your buffer is filled with HelloWorld, the loop copies 'H' into every location. You can use memset to do the same thing for you:

memset(str, 'H', 10000 * n);

The moral of the story is to avoid recalculating things that you can keep track of with a simple counter or pointer. If you want things to go fast, do as little work as possible.

Iterate C char array of size 100000 takes huge time

2 Answers2