It is what it is. I would say, at least MS gave a supported way to open external url compared to window.open()
. It's in early stage, expecting more parameters in future for openUrlOptions.
var url = "http://google.com";
var openUrlOptions = {
height: 400,
width: 800
};
Xrm.Navigation.openUrl(url, openUrlOptions);
For now, try some browser settings like below for workaround.

We are also facing behavioral discrepancy with Unified interface compared to classic.
If you are opening Entity Form or Web resource, try Xrm.Navigation.openForm
or Xrm.Navigation.openWebResource
which has boolean property openInNewWindow
under entityFormOptions
& windowOptions
parameter respectively.
Read more