The key size corresponds to a security strength. The security strength is how much effort the attacker needs to break the algorithm, and it depends upon the best attacks know against the algorithm.
For instance, a typical security strength is 128-bits. If you read a little about elliptic curve cryptography (ECC), you will learn that the best algorithm to attack ECC in the general case has square root running time. So if we use elliptic curves having prime order subgroup of 256-bit, then the number of points on the curve is order 2^256, which implies that attacking it takes sqrt(2^256) = 2^128 running time. So 256-bit ECC (having subgroup size 256-bits and key size 256-bits) gives you 128-bit security strength.
For RSA, the mathematics is harder, because the running time to break it depends upon the number field sieve, which has running time looking like e^[(1.92 + o(1)) (log n)^(1/3) * (log log n)^(2/3)], where n is the number to be factored and logs are natural logarithms. So to compute the key size corresponding to 128-bit security, we need to solve 2^128 = e^[(1.92 + o(1)) (log n)^(1/3) * (log log n)^(2/3)]. I think you would enjoy solving that yourself :-)
Note: when I am talking about security strength, I am only talking about the amount of computing power to break it. Some people argue that we should include memory as well in our calculations.
If you want a more thorough detailed discussion, I welcome you to read Selecting Cryptographic Key Sizes.
In order to select a key size, you need to understand the effort to break the algorithm, which means you should be a cryptographer. I'm sorry to tell you this, but your algorithm cannot be saved by a large key size: substitution ciphers are trivially breakable by frequency analysis.