Upgrade of WiX4 generated MSI does not work

Question

I have a WiX MSI project that is upgraded from WiX3 to WiX4 using HeatWave. When I install the MSI it works fine, but when I want to upgrade the installed MSI with a new version generated with WiX4 it does not work and the following dialog is displayed:

I suspect it has something to do with the ProductCode attribute of the Package element.

The original WiX3 wxs file is as follows:

<?xml version="1.0" encoding="UTF-8"?>
<Wix xmlns="http://schemas.microsoft.com/wix/2006/wi"
     xmlns:netfx="http://schemas.microsoft.com/wix/NetFxExtension">

    <Product Id="$(var.GuidProductId)"
             Language="1043"
             Manufacturer="Me"
             Name="$(var.ProductName)"
             UpgradeCode="$(var.GuidProductUpgradeCode)"
             Version="$(var.VersionNumber)">
        <Package Id="*"
                 Compressed="yes"
                 Description="$(var.ProductName) version $(var.VersionNumber)"
                 InstallerVersion="405"
                 InstallPrivileges="elevated"
                 InstallScope="perMachine"
                 Manufacturer="$(var.CompanyName)"
                 ReadOnly="yes"
                 ShortNames="no"
                 SummaryCodepage="Windows-1252" />
        <MajorUpgrade AllowDowngrades="no"
                      AllowSameVersionUpgrades="no"
                      Disallow="no"
                      DowngradeErrorMessage="$(var.DowngradeErrorMessage)"
                      IgnoreRemoveFailure="no"
                      Schedule="afterInstallExecute" />

        <!-- Omitted the media definition, package contents and features for brevity -->

        <!-- UI definition -->
        <Property Id="WIXUI_INSTALLDIR" Value="ROYALTIESDIR" />
        <WixVariable Id="WixUILicenseRtf" Value="license.rtf" />
        <UIRef Id="WixUI_InstallDir" />
        <UIRef Id="WixUI_ErrorProgressText" />
    </Product>
</Wix>

The updated WiX4 wxs file is as follows:

<Wix xmlns="http://wixtoolset.org/schemas/v4/wxs" xmlns:ui="http://wixtoolset.org/schemas/v4/wxs/ui">
    <Package InstallerVersion="500"
             Language="1043"
             Manufacturer="Me"
             Name="$(var.ProductName)"
             ProductCode="$(var.GuidProductId)"
             ShortNames="no"
             UpgradeCode="$(var.GuidProductUpgradeCode)"
             Version="$(var.VersionNumber)">
        <SummaryInformation Codepage="Windows-1252"
                            Description="$(var.ProductName) version $(var.VersionNumber)"
                            Manufacturer="$(var.CompanyName)" />
        <MajorUpgrade AllowDowngrades="no"
                      AllowSameVersionUpgrades="no"
                      Disallow="no"
                      DowngradeErrorMessage="$(var.DowngradeErrorMessage)"
                      IgnoreRemoveFailure="no"
                      Schedule="afterInstallExecute" />

        <!-- Omitted the media definition, package contents and features for brevity -->

        <!-- UI definition -->
        <Property Id="WIXUI_INSTALLDIR"
                  Value="ROYALTIESDIR" />
        <WixVariable Id="WixUILicenseRtf"
                     Value="license.rtf" />
        <ui:WixUI Id="WixUI_InstallDir" />
        <UIRef Id="WixUI_ErrorProgressText" />
    </Package>
</Wix>

I perform the following steps:

1. Build the installer with version 1.0
2. Install the application using this installer
3. Build the installer with version 2.0
4. Try to update the application using this new installer
5. The result is the shown dialog and no upgrade is performed.

Now when I change the value of the ProductCode attribute of the Package element to *, the application correctly upgrades, however in that situation the license agreement and directory selection screens are still shown during the upgrade. When performing the same steps with the MSI's generated with WiX3, the upgrades work correctly AND when I upgrade, nor the license agreement screen nor the directory selection screen are show. This is correct, as I do not want to show these screens when upgrading.

Now I have the following questions:

• What is the replacement in WiX4 for the Id attribute of the Product element, where I set the product id?
• How can I prevent the license agreement and directory selection screens from being displayed when I upgrade an application?

UPDATE After this error, I noticed that the referenced version numbers of the packages WixToolset.MetFx.wixext, WixToolset.UI.wixext, and WixToolset.Util.wixext were still pointing to the 4.0.0 RC4 versions. I updated these references to the latest official 4.0.0 release. Now, this dialog is gone, but upgrading is still failing.

The symptoms are as follows when I try to upgrade:

• When I start the upgrade, I get a dialog about "Resuming" the installation, but instead I am upgrading.
• Then after some progress, I get a dialog saying that the installation is has finished. However, when I look at the installation folder actually all the files installed by the previous version are gone and no installed files are present.
• Then the bundle indicates that the installation has failed, in a small not saying the same as in the initial screen shot.

In the installation log I see the following line:

[1470:269C][2023-06-02T09:53:51]i052: Condition 'WixStdBAUpdateAvailable' evaluates to false.

To me this is the indication that the installer is not able to detect it is handling an update and instead tries to install it as a regular installation.

So, one question is no longer relevant: the license agreement and directory selection screens are no longer shown with the 4.0.0 release versions. However, how to correctly detect an upgrade in my WiX4 file?