"Prevent installation of removable devices" is part of "Administrative Templates" and therefore corresponds to a Registry key.
To find out that registry key, I searched for "Prevent installation of removable devices" within C:\Windows\PolicyDefinitions\en-US\, found <string id="DeviceInstall_Removable_Deny">Prevent installation of removable devices</string> in C:\Windows\PolicyDefinitions\en-US\DeviceInstallation.adml. Then I searched for DeviceInstall_Removable_Deny in C:\Windows\PolicyDefinitions\ and found it in C:\Windows\PolicyDefinitions\DeviceInstallation.admx. The registry key is:
HKEY_LOCAL_MACHINE\Software\Policies\Microsoft\Windows\DeviceInstall\Restrictions
valueName: DenyRemovableDevices
enabled Value: decimal 1
disabled Value: decimal 0
If you change that value, it will be overridden when the group policy is applied the next time -- most likely at boot time.
According to http://learnthat.com/prevent-group-policy-from-applying-to-your-computer/ you can avoid this by denying write permissions to that specific registry key. (Note, I did not test this!)
To change the registry offline, you can boot from a windows install CD, press Shift+F10 simultaneously to open cmd, type regedit, select HKEY_LOCAL_MACHINE key, click Menu File -> Load Hive..., navigate to your installations \Windows\System32\Config\ folder and select the file named SOFTWARE.
Chose any key name, that doesn't already exist (e.g. Offline_Software) and then change the registry under that key. (e.g. HKEY_LOCAL_MACHINE\Offline_Software\Policies\Microsoft\Windows\DeviceInstall\Restrictions)
When you're done, select your loaded Hive (e.g. HKEY_LOCAL_MACHINE\Offline_Software) and click File -> Unload Hive...
To shutdown properly, just close all windows, including the setup window.
