If I am understanding you correctly, what you are trying to do is determine the relative position of two different points after a series of transforms. Working out the equations for doing this can be tricky, I find it helpful to write everything out in terms of the matrices first and then get the equations by multiplying out the matrices myself.
Lets say your anchor is at the origin, using column major matrices (like open GL, and CA Layer do). Position1 will represent the left edge of your rectangle, Position2 the right.
Position1 = {1,0,0,0,
0,1,0,0,
0,0,1,0,
0,0,0,1};
Position2 ={1,0,0,w,
0,1,0,0,
0,0,1,0,
0,0,0,1};
Position 2 now needs to be rotated by your angle around the y axis so we declare
Rotation = {cos,0,sin,0,
0,1, 0,0,
-sin,0,cos,0,
0,0, 0,1};
The rotation needs to be applied to Position2 and both of them need to be multiplied by a projection matrix. Position1 needs no rotation because it lies on the axis of rotation. It is important to realize that just setting m34 to a small negative number is not doing anything magic, what you are really doing is multiplying by the projection matrix
YourProjection = {1,0,0 ,0,
0,1,0 ,0,
0,0,1 ,0,
0,0,-1.0/1800,1};
Multiplying by a different format of projection matrix works just as well, use whatever you are most familiar with. You could use a matrix like Open GL or the kind you are using because they are really equivalent.
Position2 = MultiplyMatrices(Rotation,Position2);
Position2 = MultiplyMatrices(YourProjection,Position2);
Position1 = MultiplyMatrices(YourProjection,Position1);
Position1 and Position2 now hold the exact positions that they will be given when your CALayer performs its transform on them. Since these are in homogeneous coordinates you have to make sure to divide your x,y,z components (m41,42,43) by w (m44) to convert back to cartesian. Then all you need to do to get the width is
float width = Position1.m41-Position2.m41;
If you haven't already, you should probably make yourself functions that do matrix multiplication so that you can do calculations like this quickly and easily. Also you can't get confused because of algebra mistakes this way :). If you really want to do it algebraically just multiply this out and it should give you what you are looking for. Also in case my matrix notation was confusing, here is a conversion from column major to CALayer variables.
{m11,m12,m13,m14,
m21,m22,m23,m24,
m31,m32,m33,m34,
m41,m42,m43,m44};
where a simple translation matrix would be
{1,0,0,0,
0,1,0,0,
0,0,1,0,
x,y,z,1};