Living organisms, including plants, are very complicated miniature chemistry factories. Even separated, dead body parts still have chemical processes taking place completely independent from any parasitic organisms (bacteria, molds) present. But of course, lots of the processes which take place in the living plant don't take place any longer, and their absence can change the food considerably (rigor mortis in animals, wilting in plants).
Refrigeration slows down the growth of bacteria and molds, and some of the processes going on internally in the plants. There are two problems with that: 1) sometimes, you want these processes to go on. A pear will continue to ripen and enhance its aroma after plucking, especially if there are catalysts like ethylene present. If refrigerated, this ripening will be limited severely. 2) Some of the chemicals already present in plants will change under low temperatures. ESultanik pointed out two such examples, the self-destroying aromatic compound in tomatoes, and the starch-sugar conversion in potatoes. These and similar changes happen in other fruit and vegetables too. But it doesn't mean that all fruit and veggies undergo undesirable changes.
There is a good rule of thumb to predict where such changes may occur. It says that the bigger the temperature difference between the plant's living conditions and storage conditions, the more likely it is for its chemicals to behave strangely. So everything which thrives north of the Alps should do well in the fridge (if the fridge has a crisper, use it). Everything else must stay outside (including tomatoes - you can grow them in Middle Europe, but they don't taste well). Shape, color, etc. is quite irrelevant to storage temperature. If you have logistic difficulties, just use small breathable containers for placing berries etc. into the fridge, instead of the squishable plastic bags you may have bought them in.
The rule isn't perfect, but to do better than that, you'd have to research the optimal storage conditions for every single fruit or vegetable and possibly furnish a cold storage cabinet or pantry in the 10°C - 15°C range.