After some further source browsing and experimentation, I figured out how to force the evil approach through WixSharp. I also later discovered that this was somewhat covered by the WixSharp Sample "CustomAttributes".
"Evil Way"
File LibToReg = new File("..\Path\To\LibToReg.dll");
LibToReg.AttributesDefinition += "SelfRegCost=1";
Alternatively (based on the CustomAttributes sample):
File LibToReg = new File("..\Path\To\LibToReg.dll")
{
Attributes = new Attributes() { { "SelfRegCost", "1" } }
};
This will generate the following wxs underneath:
<Component Id="Component.LibToReg.dll" Guid="EABD7A49-26DD-4720-AE5A-AA9EEFD8C91A">
<File Id="File.LibToReg.dll" Source="..\Path\To\LibToReg.dll" SelfRegCost="1" />
</Component>
The rest of the generated code looks the same as any other DLL that is installed.
For reference, here is the original wxs source that was generated from the original Setup Project using "VDProj to WiX Converter" from Add-In Express. I believe the SelfRegCost="0" was added by the converter, but a co-worker may have manually added it in afterwards.
<Component Id='com_FB7105EC_5352_4561_AE01_405562F0EA1E' Guid='6718170E-0335-4FD6-A1E8-D9E926DDE3EC' Permanent='no' SharedDllRefCount='no' Transitive='no'>
<File Id='_FB7105EC_5352_4561_AE01_405562F0EA1E' DiskId='1' Hidden='no' ReadOnly='no' SelfRegCost='0' System='no' Vital='yes' Compressed='yes' Name='LibToReg.dll' Source='..\Path\To\LibToReg.dll' KeyPath='yes' />
</Component>