Transparency (I assume Color-to-apha) works... The "contract" is that after removal of a given color, you can put the resulting image over a background filled with that removed color and you will get back the original image.
The trouble is that this operation is ambiguous, a light gray can be obtained as a rather transparent black over white, or as a fully opaque light gray, or as slightly transparent dark gray. Or light red over dark red as a semi-transparent white or as a completely opaque light red, if you take my drift. So you have an infinity of solutions... and Gimp takes the solution that is the most transparent.
So, if you don't want some element of your picture to be included in the Color-To-Alpha, just use a selection to exclude them (but this selection should include the border pixels if you want smooth lines in the result).