[wix-users] signtool problems for 64-bit CustomAction DLLs

[Hoover, Jacob]

Which version of Wix 3 are you using, as https://github.com/wixtoolset/wix3/commit/448bd5a55d18f4ab1ca8f1cd7a0f15cf94fab88e seems to indicate that the sfxca should no longer be signed.

From: wix-users <wix-users-bounces at lists.wixtoolset.org> On Behalf Of Kidd, Nelson via wix-users
Sent: Saturday, June 11, 2022 1:04 PM
To: wix-users at lists.wixtoolset.org
Cc: Kidd, Nelson <nelson.kidd at intel.com>
Subject: [wix-users] signtool problems for 64-bit CustomAction DLLs

Hello:

I'm having trouble signing a CustomAction DLL. The unsigned CustomAction DLL functionally works within my .msi installer. I'm simply unable to digitally sign that CustomAction DLL.

This thread (https://github.com/wixtoolset/issues/issues/6089<https://github.com/wixtoolset/issues/issues/6089>) suggests that using an unsigned version of sfxca.dll should resolve the problem. I used signtool.exe to remove the existing digital signature from the 64-bit sfxca.dll, and I confirmed signature removal succeeded by viewing the file's properties in Windows Explorer. However signtool.exe still gives me the following error.

C:\_src\endurancegaming.egr547b\product\_ReleasePackage>SignTool sign /f cert.pfx /p pw InstallerCustomActions.CA.dll
Done Adding Additional Store
SignTool Error: SignedCode::Sign returned error: 0x800700C1
For more information, please see https://aka.ms/badexeformat<https://aka.ms/badexeformat>
SignTool Error: An error occurred while attempting to sign: InstallerCustomActions.CA.dll

Number of errors: 1

This thread (https://social.msdn.microsoft.com/Forums/windows/en-US/158b5381-f257-4076-83ca-db71cd8a323b/why-signtool-return-error-0x800700c1?forum=wdk<https://social.msdn.microsoft.com/Forums/windows/en-US/158b5381-f257-4076-83ca-db71cd8a323b/why-signtool-return-error-0x800700c1?forum=wdk>) suggests that error code indicates a incorrectly formatted DLL.

I used "CFF Explorer VIII" to open InstallerCustomActions.CA.dll. I observed that the [File Header]\[Machine] field is 0x14C/Intel386 and [Optional Header]\[Magic] field is 0x010B/PE32. These seem suspicious since I'm building a 64-bit DLL for a 64-bit installer.

This is my makesfxca call, and I've listed what CFF Explorer indicates for the Machine/Magic fields. 0x8664 maps to AMD64 and 0x020B maps to PE64 in CFF Explorer.

"$(WIX)\SDK\MakeSfxCA" ^
$(TargetDir)\InstallerCustomActions64.CA.dll ^ [0x8664 / 0x020B]
$(TargetDir)\sfxca.dll ^ [0x8664 / 0x020B]
$(TargetDir)\InstallerCustomActions.dll ^ [0x8664 / 0x020B]
$(TargetDir)\redacted.exe ^ [0x8664 / 0x020B]
$(TargetDir)\Microsoft.Deployment.WindowsInstaller.dll ^ [0x14C / 0x010B]
$(TargetDir)\redacted.dll ^ [0x8664 / 0x020B]
$(TargetDir)\redacted.sys ^ [0x8664 / 0x020B]
$(TargetDir)\CustomAction.config

I observed that the Microsoft.Deployment.WindowsInstaller.dll is also set to 0x14C/0x010B, like the InstallerCustomActions.CA.dll.

This seems to suggest that a possible root-cause for signtool.exe failures might be caused by MakeSfxCA using a 32-bit version of Microsoft.Deployment.WindowsInstaller.dll. I failed to find a 64-bit version of that file in the wix toolset installation folder.

Does the wix community have a suggestion on what I should try next? The CustomAction needs to 64-bit because of the .sys file, which I cannot change.

-Nelson

____________________________________________________________________
WiX Toolset Users Mailing List provided by FireGiant http://www.firegiant.com/<http://www.firegiant.com>
NOTE: This email was received from an external source. Please use caution when opening links or attachments in the message.