This isn't something you can calculate, you have to arrive at it from experience. There is no calculation that will give you a correct result, the system is just too messy to be described in simple variables.
For a rough estimate, simply use a well-written recipe that specifies the size of tin in which it will be baked. Then bake it a few times and note down the distribution of heights you get. From there:
- if you want a different height calculate the ratio of desired/actual height and scale the amount of your batter.
- If you want a different tin size, calculate the ratio of new/old bottom area and scale the amount of the batter.
Note that this won't give you the exact new height, because there isn't a linear connection between the rise of different depths of batter, but it's a very good approximation. But with a few bakes at the new size, you will get comfortable in knowing how tall it will be on average.
Also, if a very nice and even appearance is important to you, you will likely have to shave off the cake layer's top, which frequently has doming and other imperfections. So plan for baking a taller layer than you'll actually use.
Being very exact is only possible if you manage to get your process so tight down that you can reproduce it perfectly each time. This is usually only doable if you are a bakery which makes dozens of cakes per day, not for a home baker who bakes a cake per week at most.
