Because square brackets cannot be inside round brackets, every sequence of round brackets has to be complete (i.e. the count of the number of open minus close brackets is back down to zero) before each occurance of a square bracket.
1 012 1 0
()[(())()][[()()]((()))]()
10 121010 1010 123210 10
This effectively means that the square brackets split the round brackets into seperate independent sequences:
1 0 1 2 1 0
...[...]...[...[...]...]...
where complete sequences of round brackets can be placed anywhere between the square brackets.
As mentioned by Paul Hankin, the number of valid sequences of just one type of brackets is given by the Catalan Number. This gives you the number of valid sequences for the square brackets, and the number of valid sequences for any number of round brackets to be inserted somewhere inbetween the square brackets.
Unfortunately, finding out in what ways the round brackets can be distributed to the different locations inbetween the square brackets is a partitioning problem, and the logical way to solve this is by recursion. So this train of thought could lead to an algorithm, but not to one which avoids recursion.
However, the number of recursions needed to find partitions is much smaller than the Catalan number (e.g. 20 has 627 partitions, but the Catalan number for 20 is 6,564,120,420), so this method will be much faster than recursively enumerating every valid sequence.
Here's a simple example to demonstrate how this would work:
X=2, Y=3
The valid sequences of square brackets:
CatalanNumber(2) = 2
1. [ ] [ ]
2. [ [ ] ]
The positions for the round brackets:
2 × X + 1 = 5
1[2]3[4]5
1[2[3]4]5
The partitions of the round brackets:
p(Y) = 3
3
2,1
1,1,1
The permutations of the partitions:
partition: 3 (number of parts: 1)
→ 5 = 5
3,0,0,0,0 0,3,0,0,0 0,0,3,0,0 0,0,0,3,0 0,0,0,0,3
partition: 2,1 (number of parts: 2)
→ 5 × 4 = 20
2,1,0,0,0 2,0,1,0,0 2,0,0,1,0 2,0,0,0,1
1,2,0,0,0 0,2,1,0,0 0,2,0,1,0 0,2,0,0,1
1,0,2,0,0 0,1,2,0,0 0,0,2,1,0 0,0,2,0,1
1,0,0,2,0 0,1,0,2,0 0,0,1,2,0 0,0,0,2,1
1,0,0,0,2 0,1,0,0,2 0,0,1,0,2 0,0,0,1,2
partition: 1,1,1 (number of parts: 3, identical: 3)
→ (5 × 4 × 3) / (1 × 2 × 3) = 10
1,1,1,0,0 1,1,0,1,0 1,1,0,0,1
1,0,1,1,0 1,0,1,0,1 1,0,0,1,1
0,1,1,1,0 0,1,1,0,1 0,1,0,1,1 0,0,1,1,1
The valid sequences of round brackets:
CatalanNumber(3) = 5
1. ()()()
2. ()(())
3. (())()
4. (()())
5. ((()))
CatalanNumber(2) = 2
1. ()()
2. (())
CatalanNumber(1) = 1
1. ()
CatalanNumber(0) = 1
1.
The valid sequences per partition:
3,0,0,0,0 5 x 1 x 1 x 1 x 1 = 5
2,1,0,0,0 2 x 1 x 1 x 1 x 1 = 2
1,1,1,0,0 1 x 1 x 1 x 1 x 1 = 1
The total number of valid sequences:
Square brackets: 2
Round Brackets: 75
5 x 5 = 25
20 x 2 = 40
10 x 1 = 10
Total: 2 × 75 = 150