Well, the first problem is that you're trying to make a PRG from a four-byte key. This is going to be inherently insecure. An attacker will be able to guess very easily.
The second problem is that you only use ten rounds of the PRG, which means that even if you had a 128-bit key, you would only ever use the first 10, making this inherently insecure.
Your third problem is that you're using i % key.length
as the index of your mod function. This is an issue, as the index you're using will increment by exactly one. This would simulate randomness better if you were to have a larger key, but since your key is only four bytes, it won't.
Additionally, you seem to have arbitrarily added a % N
to the end of your function. I'm not sure what you're trying to accomplish, but I'm not surprised you are finding repeating patterns. You've collapsed the function down to a tiny number. It's also important to note that as your iteration count increases, your output size increases, which is generally never useful.
Your algorithm also ignores most of the key points of the RC4 generation algorithm; read here for more info.
There's no way to tell you how to fix this, because I don't think you can fix this. I think this concept is inherently insecure.