I agree those new plants don't look like rosemary. Could leave them for a little and see what happens or prune them out, but if you leave them don't wait too long or their roots may get tangled with the rosemary's.
For your rosemary plant I had a similar situation and in my case it was primarily lack of light. Over-watering is also a common problem with indoor plants as people often see plants struggling and try to give it more water to help, but if the soil is never able to try the roots will basically drown and it's very difficult for plants to recover from. Better to let soil get dry on the top 1/2 inch even if the plant looks sad, as they can recover easier from a brief drought than flooding.
Since the main issue with our plant was light, we moved it to a brighter spot. Still no effect. Eventually, after months of our rosemary looking much like yours, we setup grow lights for leafy greens and seedlings. We put the rosemary under these grow lights and pruned its branches back hard as per advice we got from a farmer friend with an exceptional green thumb. We pretty much lopped all the dead, dried branches off so that there was just a 1-2inch stump coming out of the soil. The idea is, even if the plant has ceased to grow, there may be some life left in the roots if they are given another chance. We watered it conservatively and after a few weeks under the brighter light and pruned conditions, it started sprouting! Months later it is still growing and recovering, though most new growth is only on one side for whatever reason.
In summary I recommend the following for that plant:
Ensure it has enough light.
Water it conservatively; wait until the top 1/2inch (your fingertip depth) of soil is dry, then water it through, then wait a while again.
Prune back the old dead branches to make room for new life!