The problem is two fold.
First is that thawing and unthawing make more and more water to penetrate to "object" breaking the cells. IMO such food (meat or veggies) become woody in texture and making it less tasty (sometimes I would even say I taste the freezer ice not the food).
Second is more serious. Bacteria. There are some bacteria, especially in your freezer, that develop once the food is thawing (or when the temp rise above certain degree). Such bacteria cannot be killed by re-freezing so each time you add more to the food. Same bacteria might not be killed by cooking.
NOTE: a food that have been once defrozen then properly prepared (cooked) can be refrozen again.