I do note that this definition is a bit more constrained than the usual one: in the standard definition requirement 5 is not present, and red nodes can be siblings. But given this extra restriction we can make the following analysis:
- The only tree that has a black-height of 0 is the empty tree.
- The only tree that has a black-height of 1 is a single black node (root & leaf)
The first tree that comes to mind that has a black-height of 2 is the following one:
b
/ \
b b
We can choose one of those two leaves to expand into a red node with two black children:
b
/ \
R b
/ \
b b
We cannot do more. There is no way to inject a second red node. The mirror of this tree is the only other possibility to stay within the constraints you have listed, and have a black-height of 2.
This shape is a key in incrementing to the next black-height of 3 (and greater). I will call this tree T.
To produce a tree with a black-height of 3, with as many red nodes as possible, we can replace each leaf (there are 3) with a copy of T. The result is this:
_____ b_____
/ \
R b
/ \ / \
b b R b
/ \ / \ / \
R b R b b b
/ \ / \
b b b b
Again, we can mirror the different subtrees, but there is no way to get more red nodes in a tree with a black-height of 3.
Now consider this: in this expansion we replaced each existing leaf with T, and since T has one red node, we actually added the same number of red nodes as there were leaves. And since T has 3 leaves, we multiplied the number of leaves by 3, which will determine how many T trees we will inject at the next expansion... etc.
This gives us a recurrence relation -- I write rh to represent the maximum number of red nodes in a tree of black height h:
- r0 = 0
- r1 = 0
- r2 = 1
- rh = 1 + 3rh-1
In a direct formula that last equation becomes:
Such sums can be rewritten as a repetition of 1 digits in base 3 notation, and so we can derive:
This formula gives the correct result also for h=1 and h=2, so we only need to add the case for h=0:
- r0 = 0
- rh = (3h-1 - 1) / 2, when h > 0
Here is a table with the first few results:
black height (h) | max number of red nodes (r[h])
-----------------+----------------------------------------
0 | 0
1 | 0
2 | 1
3 | 4
4 | 13
5 | 40
6 | 121
7 | 364
8 | 1093
... | ...
h | (3^(h-1) - 1) / 2