It might be anyone of a number of things - ambient interference, other WLAN's, AP firmware, Client Nic\driver\OS ... Some specific details of what your environment is running on would help.
In general though the most likely issue is interference from nearby WLAN's broadcasting in the same channels or nearby channels. There is a lot of overlap in 2.4Ghz Wifi channels. The specific 802.11 protocol you are using will also influence this - plain old 802.11b is slow but it is easier to isolate the channels it uses from those of other nearby WiFi sources or environmental RF interference from things like microwave ovens. 802.11g suffers quite badly from interference and 802.11n operating in the 2.4Ghz band can also behave erratically in the presence of competing WiFi sources, especially if you have set it up for dual channel bonding. If you have the option of using either 802.11a or 802.11n in the 5Ghz band then you will have far fewer issues with either ambient interference or interference from other nearby WiFi sources because it is much rarer and there are simply far more channels available (24 non overlapping channels vs 3 2.4Ghz).
For 2.4Ghz try fixing your channel to channel 1 and then testing. Move that to Channel 6 if you still have problems, and then try Channel 11. If you still have issues then limit both your AP and client to 802.11b and try again, if you find a satisfactory setup then try selectively enabling the faster protocols. For 802.11n set the channel width to 20Mhz first and then go higher if that is stable. Disable WMM if it's present - it doesn't help with any media streaming I've ever come across and if its enabled many WiFi Nic's get confused by the additional beaconing that it causes.
Your Client OS and adapter settings are also worth looking into. For Intel Wireless adapters disable power saving and turn Roaming Aggressiveness down to "Lowest". In general for any options that look like performance\throughput enhancements first disable all of them and then selectively enable them to see if any one works.
The dBm measurements are useful but not as important as knowing the level of ambient RF noise and the channels used by other WiFi sources. The MetaGeek WiSpy is a good way to get an idea of what is going on without spending a fortune but it is still not what you'd call cheap ($199). Professional level RF scanning costs quite a bit more.