[wix-users] signtool problems for 64-bit CustomAction DLLs

[Kidd, Nelson]

Hello:

I'm having trouble signing a CustomAction DLL. The unsigned CustomAction DLL functionally works within my .msi installer. I'm simply unable to digitally sign that CustomAction DLL.

This thread (https://github.com/wixtoolset/issues/issues/6089) suggests that using an unsigned version of sfxca.dll should resolve the problem. I used signtool.exe to remove the existing digital signature from the 64-bit sfxca.dll, and I confirmed signature removal succeeded by viewing the file's properties in Windows Explorer. However signtool.exe still gives me the following error.

C:\_src\endurancegaming.egr547b\product\_ReleasePackage>SignTool sign /f cert.pfx /p pw InstallerCustomActions.CA.dll
Done Adding Additional Store
SignTool Error: SignedCode::Sign returned error: 0x800700C1
        For more information, please see https://aka.ms/badexeformat
SignTool Error: An error occurred while attempting to sign: InstallerCustomActions.CA.dll

Number of errors: 1

This thread (https://social.msdn.microsoft.com/Forums/windows/en-US/158b5381-f257-4076-83ca-db71cd8a323b/why-signtool-return-error-0x800700c1?forum=wdk) suggests that error code indicates a incorrectly formatted DLL.

I used "CFF Explorer VIII" to open InstallerCustomActions.CA.dll. I observed that the [File Header]\[Machine] field is 0x14C/Intel386 and [Optional Header]\[Magic] field is 0x010B/PE32. These seem suspicious since I'm building a 64-bit DLL for a 64-bit installer.

This is my makesfxca call, and I've listed what CFF Explorer indicates for the Machine/Magic fields. 0x8664 maps to AMD64 and 0x020B maps to PE64 in CFF Explorer.

"$(WIX)\SDK\MakeSfxCA" ^
     $(TargetDir)\InstallerCustomActions64.CA.dll ^                                        [0x8664 / 0x020B]
     $(TargetDir)\sfxca.dll ^                                                                                   [0x8664 / 0x020B]
     $(TargetDir)\InstallerCustomActions.dll ^                                                  [0x8664 / 0x020B]
     $(TargetDir)\redacted.exe ^                                                                          [0x8664 / 0x020B]
     $(TargetDir)\Microsoft.Deployment.WindowsInstaller.dll ^                   [0x14C / 0x010B]
     $(TargetDir)\redacted.dll ^                                                                            [0x8664 / 0x020B]
     $(TargetDir)\redacted.sys ^                                                                           [0x8664 / 0x020B]
     $(TargetDir)\CustomAction.config

I observed that the Microsoft.Deployment.WindowsInstaller.dll is also set to 0x14C/0x010B, like the InstallerCustomActions.CA.dll.

This seems to suggest that a possible root-cause for signtool.exe failures might be caused by MakeSfxCA using a 32-bit version of Microsoft.Deployment.WindowsInstaller.dll. I failed to find a 64-bit version of that file in the wix toolset installation folder.

Does the wix community have a suggestion on what I should try next? The CustomAction needs to 64-bit because of the .sys file, which I cannot change.

-Nelson