Incest, well, not really, is it. No, it doesn't happen, well, not real degenerative changes, in the manner that happens with humans if they breed with close relatives, and not in plants like calendula grown outdoors where they are in the open and free to reproduce as they will. Sometimes there may be slight changes - perhaps a flower arrives on, say, a dahlia or some other flowering plant which is different in some way from all the others. It may have a colour break where all the others are one colour, or maybe the petals might be somewhat frilly when the others aren't - when this happens, breeders may select that one and try to breed from it in hopes of creating a new variety of a particular plant.
Where trouble occurs is where it where it always occurs in both flora and fauna - when breeding is restricted, usually by man, under controlled conditions, and plants such as F1 hybrids are created. Many hybrids don't have good vigour, or don't retain the characteristics they were bred for, or are, in fact, sterile. You might call this in breeding, which is a term that can be applied to humans and animals too, and it often leads to significant problems.
There is, though, a good level of success with crossing and recrossing plants to get new varieties, but it's trial and error. Many 'faulty' crossed versions of plants will have been rejected before one is finally selected as being good enough to put on sale.
In the case of your 'bed of calendulas' example, environmental conditions will usually be the culprit - the soil is exhausted, or there's a build up of pathogens/insects which affect that particular plant, a common problem with mono culture of vegetables.