Adding a new Custom Action to a project prevents an existing Custom Action from being run

Question

I have a Custom Action project that has various CA's used by installers that my company creates, a few of those are used to manipulate the IIs7 through the Microsoft.Web.Administration API.

I added a new custom action called SetApplicationAutoStart the the class containing IIs related CA's. This custom action is used to set the autoStart attribute that forces the IIs to preload and start WCF services so that initial response time will be shorter.

After adding this action an existing CA called SetAppPoolLoadUserProfileTrue stopped working. This CA forces that setting on a site to true, even if the default site on the computer has been changed so that this setting is false, so we really need it to work.

The log files contains the following lines when the action fails.

MSI (s) (A0:18) [15:02:43:639]: Executing op: ActionStart(Name=SetAppPoolLoadUserProfileTrue,,)
Action 15:02:43: SetAppPoolLoadUserProfileTrue. 
MSI (s) (A0:18) [15:02:43:641]: Executing op: CustomActionSchedule(Action=SetAppPoolLoadUserProfileTrue,ActionType=3073,Source=BinaryData,Target=SetAppPoolLoadUserProfileTrue,CustomActionData=AppPoolName=xxxxx)
MSI (s) (A0:18) [15:02:43:670]: Creating MSIHANDLE (377) of type 790536 for thread 50712
MSI (s) (A0:C8) [15:02:43:670]: Invoking remote custom action. DLL: C:\Windows\Installer\MSIBD82.tmp, Entrypoint: SetAppPoolLoadUserProfileTrue
CustomAction SetAppPoolLoadUserProfileTrue returned actual error code 1154 (note this may not be 100% accurate if translation happened inside sandbox)
MSI (s) (A0:C8) [15:02:43:673]: Closing MSIHANDLE (377) of type 790536 for thread 50712
MSI (s) (A0:18) [15:02:43:674]: Note: 1: 1723 2: SetAppPoolLoadUserProfileTrue 3: SetAppPoolLoadUserProfileTrue 4: C:\Windows\Installer\MSIBD82.tmp 
Error 1723. There is a problem with this Windows Installer package. A DLL required for this install to complete could not be run. Contact your support personnel or package vendor.  Action SetAppPoolLoadUserProfileTrue, entry: SetAppPoolLoadUserProfileTrue, library: C:\Windows\Installer\MSIBD82.tmp 
MSI (s) (A0:18) [15:20:25:139]: Product: xxxxxxx -- Error 1723. There is a problem with this Windows Installer package. A DLL required for this install to complete could not be run. Contact your support personnel or package vendor.  Action SetAppPoolLoadUserProfileTrue, entry: SetAppPoolLoadUserProfileTrue, library: C:\Windows\Installer\MSIBD82.tmp 
Action ended 15:20:25: InstallFinalize. Return value 3.

This looks like a problem extracting the dotnet PE from the PE for this action. All other CA's in the binary work properly including the new one.

Be aware that Visual Studio did not preselect my CustomActions project to build when building my solution (Configuration Manager). This can lead to an unexpected out-of-date Custom Action assembly. — Adam Caviness, May 04 '16 at 15:12

galets · Answer 1 · 2014-08-04T15:07:17.007

I experienced exactly the same symptom you are describing. There seems to be a problem with WiX toolset. My version of WiX tolset is 3.8, and I also had a custom action, which would not run, and changing its name fixed the problem. The SFX compiler simply compiles the broken DLL without any indication of a problem. What makes the matters worse is that in my case this was a function which was supposed to run on uninstall with Result="ignore", so I wouldn't even have any immediate indication that there is a problem after I run actual installer.

I tried experimenting to understand what exactly is the problem with the name, and did not find any satisfactory explanation. It seems to not matter where in the source code the offending function is, what is it alphabetically (e.g.: functions succeed which are before and after it alphabetically). Loading DLL into depends.exe shows that there is an export, but trying to rundll32 it fails to find that export. Changing its name fixes the problem. Also, sometimes you can add another function, and the failing one would succeed, but the one you just added fails instead.

Here's what I have done to fix the problem: I wrote a C++ program which loads the compiled custom action and verifies the export. Then I wrote a unit test, which verified all of the exports in custom action. This obviously does not fix the issue, but at least you will have a unit test failure and know that your installer is broken. Here's the c++ code if you are interested:

typedef int(__stdcall *CustomActionProc)(HANDLE);

int _tmain(int argc, _TCHAR* argv[])
{
    if (argc != 3)
    {
        _tprintf(_T("Parameters: DLL, EntryPoint\n"));
        return 1;
    }

    LPCTSTR dllName = argv[1];
    LPCTSTR entryPoint = argv[2];

    HMODULE hLib = LoadLibrary(dllName);
    if (hLib == NULL)
    {
        _tprintf(_T("Error loading %s\n"), dllName);
        return 1;
    }

    CustomActionProc procAddress = 
        (CustomActionProc) GetProcAddress(hLib, CStringA(entryPoint));
    if (procAddress == NULL)
    {
        _tprintf(_T("Error locating entrypoint %s\n"), entryPoint);
        return 1;
    }

    return 0;
}

And the unit test is:

    [TestMethod]
    public void TestCustomActionCanBeInvoked()
    {
        var asm1 = typeof(MyCustomActionsClass).Assembly;
        var methods = asm1.GetTypes().SelectMany(t =>
            t.GetMethods().Where(m => m.GetCustomAttributes(false)
                    .Where(a => a.GetType().Name == "CustomActionAttribute").Any()));

        var binFolder = (new FileInfo(this.GetType().Assembly.Location)).DirectoryName;
        var customActionsSfx = Path.Combine(binFolder, "MyCustomAction.CA.dll");
        var testMsiExport = Path.Combine(binFolder, "TestMsiExport.exe");

        foreach (var m in methods)
        {
            Trace.WriteLine("Method Name: " + m.Name);

            var p = Process.Start(new ProcessStartInfo()
            {
                FileName = testMsiExport,
                Arguments = "\"" + customActionsSfx + "\" " + m.Name,
                UseShellExecute = false,
                RedirectStandardOutput = true,
            });

            p.OutputDataReceived += (s, d) => Trace.WriteLine(d.Data);
            p.BeginOutputReadLine();
            p.WaitForExit();

            if (p.ExitCode != 0)
            {
                Assert.Fail("Bad Sfx export detected! Export name: " + m.Name);
            }
        }
    }

Hope this helps someone in my situation. It was a very frustrating day trying to nail this down.

score 2 · Accepted Answer · edited May 23 '17 at 10:28

Same thing happened to me. "galets" was already pointing in the right direction, setting me on track (no rep to upvote, sorry).

Short version:

MakeSfxCA produces native dlls not following the PE / COFF spec leading to the observed behaviour.

Long version:

Construct a CA, e.g. "HavocAction", having three exported entry points (i.e. marked with "CustomAction" attribute), named "HavocEntryPointa", "HavocEntryPointB", "HavocZappEntryPoint" (mind the exact spelling). The methods may just return "ActionResult.Success".
Come up with simple setups, a) invoking just "HavocEntryPointa", b) invoking just "HavocEntryPointB"
==> Setup "a)" will work, Setup "b)" will fail
Uncomment the "CustomAction" attribute on "HavocZappEntryPoint" and the behaviour is inverted, i.e.
==> Setup "a)" will fail, Setup "b)" will work

Further analysis

When you dump the wrapped CA.dll-file with

dumpbin /Exports HavocAction.CA.dll

you get something like (excerpt)

125   7C 00003A36 
126   7D 00003A4C HavocEntryPointa
127   7E 00003A62 HavocEntryPointB
128   7F 00003A78 HavocZappEntryPoint
129   80 000042FC zzzEmbeddedUIHandler
130   81 000043B8 zzzInitializeEmbeddedUI
131   82 0000467F zzzShutdownEmbeddedUI
132   83 00003AA5 zzzzInvokeManagedCustomActionOutOfProcW

This is wrong (search for "pecoff_v83.docx", cf. Exported DLL functions not ordered lexically?). The entries are supposed to be sorted (by ASCII) in order to do a binary search when loading methods from the dll (the entries "HavocEntryPointa" and "HavocEntryPointB" are interchanged).

My educated guess is, when loading code from the dll the binary search fails, resulting in the error. Due to the nature of a binary search, removing "HavocZappEntryPoint" inverts the effect.

Remark regarding OP

Kjartan first used "SetApplicationAutoStart" and "SetAppPoolLoadUserProfileTrue" which was not correctly exported to the CA.dll due to wrong ordering; the upper case letter "P" comes before the lower case "l" but this was interchanged by MakeSfxCA. His latter choice "ConfigureApplicationAutoStart" and "SetAppPoolLoadUserProfileTrue" is ordered conforming to the PE / COFF spec.

PS: This is http://wixtoolset.org/issues/4502 now.

Update

PPS: Starting with the WiX 3.9 RC3 release the bug fix for this issue is included; everything works as expected.

Changed this to the correct answer since this will allow people to create the proper workarounds. — Kjartan Þór Kjartansson, Aug 22 '14 at 16:31

score 1 · Answer 3 · answered Aug 15 '12 at 18:01

1

This is actually pretty strange but after a long time searching for answers and trying lot's of different things I tried changing the name of the new CA from SetApplicationAutoStart to ConfigureApplicationAutoStart and that resulted in SetAppPoolLoadUserProfileTrue to start working properly again

answered Aug 15 '12 at 18:01

Kjartan Þór Kjartansson

555
3
10

I have also seen that a certain .DLL breaks the CA functionality, and just renaming this .DLL file gets it up and running. Very strange. – Morten Frederiksen Aug 04 '14 at 15:41

Adding a new Custom Action to a project prevents an existing Custom Action from being run

3 Answers3

Linked