Many of those are available online; that's certainly where my wife and I have obtained most of these sorts of ingredients. That's usually the "quickest" way to get them, in the sense of least effort to find something (not necessarily quickest to actually arrive of course). Citric Acid, Gum Arabic, etc. are easily available in "moderate" quantities (think a pound or so) online.
Some are also available at larger grocers, depending on exactly what you're looking for. If you're in a larger city, there are also often specialty stores that carry some of these ingredients; especially if they're frequently used in a particular culture's cuisine, for example there is a Korean market near my work that has quite a lot of specialty ingredients that I can't find easily elsewhere or even easily find online in moderate quantities. Look in the "bulk" area of the store - next to the bulk nuts and seeds and such; often you'll find other powders, like many of the gums and acids.
I'd also check with the larger spice shops - while what you're describing isn't a spice, sometimes they'll carry them anyway due to the similarities (and, probably, the similar distributors). I've found for example gum arabic at my local (larger) spice store.
