Why reimplement strlen as loop+subtraction?

Question

Inspired by this question about the following code from SQLite3:

 static int strlen30(const char *z){
    const char *z2 = z;
    while( *z2 ){ z2++; }
    return 0x3fffffff & (int)(z2 - z);
 }

that is accompanied by a commit message saying this function helps with int overflows.

I'm particularly interested in this part:

 const char *z2 = z;
 while( *z2 ){ z2++; }

to me this loop advances z2 until z2 points onto null terminator. Then z2-z yields the string length.

Why not use strlen() for this part and rewrite like this:

return 0x3fffffff & (int)(strlen(z));

Why use loop+subtraction instead of strlen()? What can loop+subtraction do what strlen() can't?

Answer 1

I can't tell you the reason why they had to re-implement it, and why they chose int instead if size_t as the return type. But about the function:

/*
 ** Compute a string length that is limited to what can be stored in
 ** lower 30 bits of a 32-bit signed integer.
 */
static int strlen30(const char *z){
    const char *z2 = z;
    while( *z2 ){ z2++; }
    return 0x3fffffff & (int)(z2 - z);
}

Standard References on Truncation, Types, Overflow

The standard says in (ISO/IEC 14882:2003(E)) 3.9.1 Fundamental Types, 4.:

Unsigned integers, declared unsigned, shall obey the laws of arithmetic modulo 2ⁿ where n is the number of bits in the value representation of that particular size of integer. 41)

...

⁴¹⁾: This implies that unsigned arithmetic does not overflow because a result that cannot be represented by the resulting unsigned integer type is reduced modulo the number that is one greater than the largest value that can be represented by the resulting unsigned integer type

That part of the standard does not define overflow-behaviour for signed integers. If we look at 5. Expressions, 5.:

If during the evaluation of an expression, the result is not mathematically defined or not in the range of representable values for its type, the behavior is undefined, unless such an expression is a constant expression (5.19), in which case the program is ill-formed. [Note: most existing implementations of C + + ignore integer overflows. Treatment of division by zero, forming a remainder using a zero divisor, and all floating point exceptions vary among machines, and is usually adjustable by a library function. ]

So far for overflow.

As for subtracting two pointers to array elements, 5.7 Additive operators, 6.:

When two pointers to elements of the same array object are subtracted, the result is the difference of the subscripts of the two array elements. The type of the result is an implementation-defined signed integral type; this type shall be the same type that is defined as ptrdiff_t in the header (18.1). [...]

Looking at 18.1:

The contents are the same as the Standard C library header stddef.h

So let's look at the C standard (I only have a copy of C99, though), 7.17 Common Definitions :

1. The types used for size_t and ptrdiff_t should not have an integer conversion rank greater than that of signed long int unless the implementation supports objects large enough to make this necessary.

No further guarantee made about ptrdiff_t. Then, Annex E (still in ISO/IEC 9899:TC2) gives the minimum magnitude for signed long int, but not a maximum:

#define LONG_MAX +2147483647

Now what are the maxima for int, the return type for sqlite - strlen30()? Let's skip the C++ quotation that forwards us to the C-standard once again, and we'll see in C99, Annex E, the minimum maximum for int:

#define INT_MAX +32767

Summary about the truncation part

1. Usually, ptrdiff_t is not bigger than signed long, which is not smaller than 32bits.
2. int is just defined to be at least 16bits long.
3. Therefore, subtracting two pointers may give a result that does not fit into the int of your platform.
4. We remember from above that for signed types, a result that does not fit yields undefined behaviour.
5. strlen30 does applies a bitwise or upon the pointer-subtract-result:

---

          | 32 bit                         |
ptr_diff  |10111101111110011110111110011111| // could be even larger
&         |00111111111111111111111111111111| // == 3FFFFFFF<sub>16</sub>
          ----------------------------------
=         |00111101111110011110111110011111| // truncated

---

That prevents undefiend behaviour by truncation of the pointer-subtraction result to a maximum value of 3FFFFFFF₁₆ = 1073741823₁₀.

I am not sure about why they chose exactly that value, because on most machines, only the most significant bit tells the signedness. It could have made sense versus the standard to choose the minimum INT_MAX, but 1073741823 is indeed slightly strange without knowing more details (though it of course perfectly does what the comment above their function says: truncate to 30bits and prevent overflow).

"Why not use strlen() for this part"

and rewrite it like this:

return 0x3fffffff & (int)(strlen(z));

My guess is that they wanted to avoid a potential indirection. Another advantage might be fewer dependencies on the standard library, which can be useful if you write a non-hosted application.

Btw, as follows from the references above, (int)(strlen(z)) might yield undefined behaviour if the maximum for ptrdiff_t > INT_MAX, so (int)(0x3fffffff & strlen(z)) would be better.

Answer 2

Why reimplement strlen as loop+subtraction?

I suspect the real answer is that the programmer felt like it, but another potential justification/rationalisation is that the loop is inline (independent of whether strlen30 itself is), whereas on many systems strlen is an out-of-line function call (e.g. Linux/GCC). If the overwhelming majority of strings are empty or short (despite the "special" treatment of long ones), then that may yield a slight performance bump for the common case. That possibility alone may be enough to get a code-happy programmer key-tapping. For longer strings I would expect the library strlen to be generally optimal (allowing for it's lack of knowledge of the application specific length of strings).

Some systems may not even benefit from this inlining as strlen provides it's own, or an inline/out-of-line hybrid with a quick inline check for empty, one-char, maybe two-char strings then a call.