Sedgewick Algorithms 4, why BinarySearchST put FrequencyCounters test costs lower than SequentialSearchST?

Question

I'm reading Algorithms 4th edition. I have some questions when reading chapter 3 Searching. From the cost summary the insert cost of BinarySearchST(2N in worst case) is a little worse than SequentialSearchST(N in worst case). But the FrequencyCounter test with VisualAccumulator(which draws plots) shows

Returning to the cost of the put() operations for FrequencyCounter for words of length 8 or more, we see a reduction in the average cost from 2,246 compares (plus array accesses) per operation for SequentialSearchST to 484 for BinarySearchST.

Shouldn't the put() operations of BinarySearchST need more compares(plus array accesses) than SequentialSearchST?

Another question, for BinarySearchST, the book says

Proposition B (continued). Inserting a new key into an ordered array of size N uses ~ 2N array accesses in the worst case, so inserting N keys into an initially empty table uses ~ N² array accesses in the worst case

When I look at the code of BinarySearchST, I think inserting a new key into an ordered array of size N uses ~ 4N array accesses.

    public void put(Key key, Value val)  {
    if (key == null) throw new IllegalArgumentException("first argument to put() is null"); 

    if (val == null) {
        delete(key);
        return;
    }

    int i = rank(key);

    // key is already in table
    if (i < n && keys[i].compareTo(key) == 0) {
        vals[i] = val;
        return;
    }

    // insert new key-value pair
    if (n == keys.length) resize(2*keys.length);

    for (int j = n; j > i; j--)  {
        keys[j] = keys[j-1];
        vals[j] = vals[j-1];
    }
    keys[i] = key;
    vals[i] = val;
    n++;

    assert check();
} 

Because for every i in the loop, there are 4 array accesses, 2 for keys reading and updating, 2 for values reading and updating. So why does prop B say it uses ~2N array accesses?

Answer 1

Shouldn't the put() operations of BinarySearchST need more compares(plus array accesses) than SequentialSearchST?

The key thing to understand is where does complexity comes from for each of these two symbol table implementations. SequentialSearchST reaches its worst case when the input key is not present, because in that case it needs to perform N searches (and has N misses). Based on the type of the input text, this could happen quite often. However, even if the key is already there, on average there are N/2 compares to find it sequentially.

As per BinarySearchST, searching for the key costs logN in the worst case, so here the complexity comes from resizing the array and/or from moving the existing elements to the right to make room for a new key. Notice that when the key is missing, you should make N/2 moves on average, and when key is there, only logN compares on average. In this case the total running time highly depends on the distribution of the keys - if new keys keep coming, running time will be higher!

The test they performed included text "Tale of two cities" by Charles Dickens, taking only words with 8 letters or more. There are 14350 such words, from which 5737 distinct. After 14350 put() operations and 5737 keys in the table, you would expect about 5737 / 2 = 2868 compares to perform another put() in SequentialSearchST. However, it's better than that, you "only" need 2246 compares. BinarySearchST's runtime significantly depends on the presence of the key; the experiment showed that for this text there were far more O(logN) searches of existing keys than O(N) moves required to insert new keys, which combined gives smaller cost than SequentialSearchST. Do not mix average and worst case runtime, this analysis relies on the average case complexity for the specific example.

When I look at the code of BinarySearchST, I think inserting a new key into an ordered array of size N uses ~ 4N array accesses.

Authors should have clarified the exact definition of access. If referencing the array element means access, then there are even more, 8N array accesses because in the worst case you should first resize the whole array (take a look at the implementation of resize()). Of course, whole implementation could be rewritten to optimize number of accesses in this case by putting new key at the right place during the resize operation.

Answer 2

"Shouldn't the put() operations of BinarySearchST need more compares(plus array accesses) than SequentialSearchST?"

No, because previously the book talks about the WORST case.

Worst and Average cases are different. From the next sentence of the book we can read : "As before, this cost is even better than would be predicted by analysis, and the extra improvement is likely again explained by properties of the application ..."

"So why prop B says it uses ~2N array accesses?"

At some point, I think, you are right, formally there are 4N accesses, but

what if we rewrite loop as :

keys[j] = keys[j-1];
keys[j-1] = keys[j-2];
keys[j-2] = keys[j-3];
...
keys[i+1] = keys[i];

will it mean that we still use 4N accesses? I assume, that JIT compiler can optimize the loop in a right way.

Also we can do an assumption that arrays usually represented as a linear memory, computers read data into virtual pages, so, such a page has been already accessed and it is in a cache.

Answer 3

If a binary search tree is "balanced", there will be far less comparisons.

1         d
        /   \
2     b       f
    /   \   /   \
3  a     c e     g

In the worst case "unbalanced", there will be more, on the same "order" as sequential. It's not a linear reduction when the tree is balanced, I think it's C * (ln(2) / ln(n+1)) or just O(log(N)) for short. So for millions of records there are much much less.

1  a
    \
2    b
      \
3      c
        \
4        d
          \
5          e
            \
6            f
              \
7              g

If it's only a little unbalanced, the result will be somewhere in the middle.

1         d
        /   \
2     b       e
    /   \       \
3  a     c        f
                   \
4                    g

I'm not sure that your code is optimal, but if the book says there are twice as many operations in the worst case, it's probably accurate. Try to get it to 2x at each level if you're interested in the details for academic reasons.

I wouldn't worry about the value of C - you probably only want to use a BST if you know in advance it's going to be balanced or close to balanced based on your insertion/update method because O(N) will probably be catastrophic. Consider 40 * (ln(2) / ln(1,000,0000,000,000+1)) versus 1 * 1,000,000,000,000.

Answer 4

The point about BinarySearchST vs SequentialSearchST performance in average case was already covered in other responses.

Concerning the second question: 2N is for an array. It’s obviously true. The BinarySearchST uses 2 arrays but anyway when you’re inserting in an initially empty tree N times you get ~N^2 operations. It’s up to a multiplier. Either you have 2 + 4 + 6 + ... + 2N or 2 times that - anyway you get ~N^2.