Yes, you should be clearing those out. That looks like a bunch of deciduous trees coming up in front of the conifers. We have the same problem with the woods in our pasture. Basically, what'll happen is the tree branches will grow out into your yard, causing you not to mow out as far, then new trees will grow up under the branches and extend further into the yard. Eventually, you start loosing your yard.
In the pasture, I just poison under the limbs and take a chainsaw to them if they start to stick out to far, which the will when reaching toward the light. However, with this being a yard, you might want to take a different approach. With the smaller amount of work, I'd just do it manually. You need to make a decision, though. I'd take out all the small trees, without doubt. The decision you have to make is with the coniferous trees. Ideally, they would have been set back far enough from your yard when planted, that when full grown, they wouldn't be hanging over into your yard and preventing you from mowing the yard.
Your two options at this point would be to either let them have that amount of yard, or limb them up to a height where they don't affect your mowing. You can trim back just the branches that are hanging over your yard, but it might not look good. You can always do that and if you don't like it, limb them back to the trunk and up to a height where they don't affect your mowing. It's really an aesthetics thing, once you know that you have to cut it. I know those trees look good when they have branches all the way to the ground, but they can also look fine limbed up a bit. So either give them some of the yard or limb them up. Definitely clean up that scrubby stuff, though.
