That depends on if you have control of both the client and the server, how you're storing/fixing the key, and how difficult it is to deploy replacements to the client/server code. If it's not that difficult for you to do a re-deploy and you know that all instances of the client will be updated promptly I suppose you could just do a re-deploy if you ever needed to rotate the key.
However, if it's not that easy for you to do a re-deploy and/or you can't guarantee that all of the code will be updated promptly this could be a problem because you could be left without a method of rotating the key if your current one became compromised.
The other problem is that you don't really want to have all of your clients using the same private key, especially if you have a lot of clients. This increases the risk of your private key eventually being compromised and makes it more difficult for the server to verify the authenticity of individual clients. Additionally, this would create a single point of failure because if the private key is compromised, it would compromise the communications of all of the clients. If each client has its own private/public key, if its private key is compromised there's no impact on other clients so the damage would be much more contained.
Using per-client keys, compromising a private key could only compromise communications for a single client that occurred since the last key rotation; if, for example, you're rotating keys every 3 months compromising a key can never compromise more than 3 months of communication for a single client. (On average, it would compromise 1.5 months of communication for that particular client).
Incidentally, maybe I'm misreading your question but it's not important with public-key cryptography that the public key remain secret, just that you have a way of protecting the private key. The .NET Framework does have key containers for the purpose of safely storing the private key.