[wix-users] signtool problems for 64-bit CustomAction DLLs

[Kidd, Nelson]

Hi Jacob,

Thanks for responding. I am using latest recommended build, WiX Toolset 3.11.2.4516... which is apparently old.

I downloaded development build WiX Toolset 3.14.0.6526, and I can confirm that sfxca.dll is not signed. I can also confirm that I'm able to sign the .ca.dll file when using the more recent development build of sfxca.dll. Based on the commit history of SfxCA folder, all files except the .vcxproj are unchanged for 6 years, so I expect the development build version is functionally identical as the recommended build.

-Nelson

From: Hoover, Jacob <Jacob.Hoover at greenheck.com>
Sent: Monday, June 13, 2022 7:00 AM
To: WiX Toolset Users Mailing List <wix-users at lists.wixtoolset.org>
Cc: Kidd, Nelson <nelson.kidd at intel.com>
Subject: RE: signtool problems for 64-bit CustomAction DLLs

Which version of Wix 3 are you using, as https://github.com/wixtoolset/wix3/commit/448bd5a55d18f4ab1ca8f1cd7a0f15cf94fab88e seems to indicate that the sfxca should no longer be signed.

From: wix-users <wix-users-bounces at lists.wixtoolset.org<mailto:wix-users-bounces at lists.wixtoolset.org>> On Behalf Of Kidd, Nelson via wix-users
Sent: Saturday, June 11, 2022 1:04 PM
To: wix-users at lists.wixtoolset.org<mailto:wix-users at lists.wixtoolset.org>
Cc: Kidd, Nelson <nelson.kidd at intel.com<mailto:nelson.kidd at intel.com>>
Subject: [wix-users] signtool problems for 64-bit CustomAction DLLs

Hello:

I'm having trouble signing a CustomAction DLL. The unsigned CustomAction DLL functionally works within my .msi installer. I'm simply unable to digitally sign that CustomAction DLL.

This thread (https://github.com/wixtoolset/issues/issues/6089) suggests that using an unsigned version of sfxca.dll should resolve the problem. I used signtool.exe to remove the existing digital signature from the 64-bit sfxca.dll, and I confirmed signature removal succeeded by viewing the file's properties in Windows Explorer. However signtool.exe still gives me the following error.

C:\_src\endurancegaming.egr547b\product\_ReleasePackage>SignTool sign /f cert.pfx /p pw InstallerCustomActions.CA.dll
Done Adding Additional Store
SignTool Error: SignedCode::Sign returned error: 0x800700C1
For more information, please see https://aka.ms/badexeformat
SignTool Error: An error occurred while attempting to sign: InstallerCustomActions.CA.dll

Number of errors: 1

This thread (https://social.msdn.microsoft.com/Forums/windows/en-US/158b5381-f257-4076-83ca-db71cd8a323b/why-signtool-return-error-0x800700c1?forum=wdk) suggests that error code indicates a incorrectly formatted DLL.

I used "CFF Explorer VIII" to open InstallerCustomActions.CA.dll. I observed that the [File Header]\[Machine] field is 0x14C/Intel386 and [Optional Header]\[Magic] field is 0x010B/PE32. These seem suspicious since I'm building a 64-bit DLL for a 64-bit installer.

This is my makesfxca call, and I've listed what CFF Explorer indicates for the Machine/Magic fields. 0x8664 maps to AMD64 and 0x020B maps to PE64 in CFF Explorer.

"$(WIX)\SDK\MakeSfxCA" ^
$(TargetDir)\InstallerCustomActions64.CA.dll ^ [0x8664 / 0x020B]
$(TargetDir)\sfxca.dll ^ [0x8664 / 0x020B]
$(TargetDir)\InstallerCustomActions.dll ^ [0x8664 / 0x020B]
$(TargetDir)\redacted.exe ^ [0x8664 / 0x020B]
$(TargetDir)\Microsoft.Deployment.WindowsInstaller.dll ^ [0x14C / 0x010B]
$(TargetDir)\redacted.dll ^ [0x8664 / 0x020B]
$(TargetDir)\redacted.sys ^ [0x8664 / 0x020B]
$(TargetDir)\CustomAction.config

I observed that the Microsoft.Deployment.WindowsInstaller.dll is also set to 0x14C/0x010B, like the InstallerCustomActions.CA.dll.

This seems to suggest that a possible root-cause for signtool.exe failures might be caused by MakeSfxCA using a 32-bit version of Microsoft.Deployment.WindowsInstaller.dll. I failed to find a 64-bit version of that file in the wix toolset installation folder.

Does the wix community have a suggestion on what I should try next? The CustomAction needs to 64-bit because of the .sys file, which I cannot change.

-Nelson

____________________________________________________________________
WiX Toolset Users Mailing List provided by FireGiant http://www.firegiant.com/<http://www.firegiant.com>
NOTE: This email was received from an external source. Please use caution when opening links or attachments in the message.