Since it's a question of opinion, I'm going to give my own as a developer but also a power smartphone user (well, who isn't nowadays):
tl;dr: leave it running
===============================
longer version
The point of using your phone as a music player, is providing you with audio while you're doing other activities, like running, browsing, texting, or even playing music for others connected through a speaker, being the "dj" of your group. You would rarely use a music player as a primary task and I would expect to do that when you're doing something like trying to figure out the lyrics, or watch the videoclip (and hence, you would use YouTube). Thus, it is my belief that your music player should have a separate lifecycle than the rest of your phone activities. Imagine the nightmare of playing music for others and music suddenly stops while you're messing with unrelated stuff on your phone.
However, you have a point when mentioning that "it seems that users will expect the audio to stop and the notification to disappear from the status bar together with the app". I wouldn't get the whole statement as true, rather extract the gist: users want to stop their music app easily.
In that sense, you should make it as easy as possible to stop playback to optimize your user experience. Out of the top of my head, I would imagine the best way of doing that would be a nice "X" button in your notification extended (or even when compact) version. The user then can stop the playback right from the status bar and not have to go through bringing the app to the front.
If you do want to go a step further, you could have an option in your settings to either use a foreground or background service -to make it easier for the user to understand, you could use wording like "stop music when recent apps are cleared", hence delegating the choice to your user, according to their needs. That, of course, would add complexity and too much power to your users so it's up to you to figure out if you need it.