One thing I have noticed is that whichever app is installed first will be the one that example.com forwards to.
You are correct to attribute this behaviour to Nginx. I think it's due to it falling back to this config somehow when it doesn't detect a config for example.com
This dokku plugin (https://github.com/progrium/dokku/tree/master/plugins/nginx-vhosts) is responsible for rewriting the nginx.conf for each app every time it is deployed.
Nowadays it uses a template nginx.conf (https://github.com/progrium/dokku/blob/master/plugins/nginx-vhosts/templates/nginx.conf) although this is a fairly recent change so be sure your on a recent version.
You will end up with a Nginx config that looks like the following:
server {
listen [::]:80;
listen 80;
server_name app.example.com;
return 301 https://$host$request_uri;
}
I'm not currently sure why the above snippet results in the described behaviour. A work around is to setup your own nginx conf in /etc/nginx/sites-enabled/ with
server_name example.com;
but pointing to a holding page or whatever works for you.