For this problem regarding Windows Firewall settings...
It turned out that settings made in Windows Settings were messing with the ADM(X) templates.
Computer Configuration
|+ Policies
|+ Windows Settings
|+ Security Settings
|+ Windows Firewall with Advanced Security - LDAP://...
Here, configuring Windows Firewall Properties for Private and Public Profile to Not configured removed everything else but Software\Policies\Microsoft\WindowsFirewall\PolicyVersion
from the Extra Registry Settings. (Now, it's of course possible to set them as wanted from here, too.)

This is good, as I checked the windowsfirewall.admx
in all Administrative Templates through Windows Vista, Windows 7 and Windows 10; there weren't any settings for the Private and Public profile: just for the Domain Profile and Standard Profile. If I didn't find this solution, it would have required using the methods explained below.
Removing Extra Registry Settings from Default Domain Policy in general
Easiest way to solve this would be to remove the GPO involved and re-create it with only the necessary settings. For Default Domain Policy this needs some extra steps:
- Print out / Save report of all your Default Domain Policy GPO settings.
Recreate the default Group Policy Object using Dcgpofix (for the domain only, not for DC):
DCGPOFix /ignoreschema /target:Domain
Edit your policy manually to contain all the settings in the report.
Other way is to manually create a new Administrative Template containing settings for these registry keys; .admx
files are XML and easy to edit with a text editor.
In this case for Windows Firewall it would have been possible to edit the windowsfirewall.admx
:
Create two new categories. (I hard-coded the displayName
s to avoid modifying any .adml
s.)

Copy all (or just the required) child policy
objects of WF_Profile_Standard
.
Replace contents as required: Standard
with Public
/ Private
:
<parentCategory ref="WF_Profile_Public" />
key="SOFTWARE\Policies\Microsoft\WindowsFirewall\PublicProfile\...
- No need to replace any
displayName
s, explainText
s or presentation
s as they are already the same for both of the existing categories.
I'd recommend using this new template only temporarily & from a client computer having the Remote Server Administration Tools installed, instead using it directly on a DC. This way, it wouldn't cause the very problems you are trying to solve with it!