[wix-users] signtool problems for 64-bit CustomAction DLLs
nelson.kidd at intel.com
Sat Jun 11 11:03:34 PDT 2022
I'm having trouble signing a CustomAction DLL. The unsigned CustomAction DLL functionally works within my .msi installer. I'm simply unable to digitally sign that CustomAction DLL.
This thread (https://github.com/wixtoolset/issues/issues/6089) suggests that using an unsigned version of sfxca.dll should resolve the problem. I used signtool.exe to remove the existing digital signature from the 64-bit sfxca.dll, and I confirmed signature removal succeeded by viewing the file's properties in Windows Explorer. However signtool.exe still gives me the following error.
C:\_src\endurancegaming.egr547b\product\_ReleasePackage>SignTool sign /f cert.pfx /p pw InstallerCustomActions.CA.dll
Done Adding Additional Store
SignTool Error: SignedCode::Sign returned error: 0x800700C1
For more information, please see https://aka.ms/badexeformat
SignTool Error: An error occurred while attempting to sign: InstallerCustomActions.CA.dll
Number of errors: 1
This thread (https://social.msdn.microsoft.com/Forums/windows/en-US/158b5381-f257-4076-83ca-db71cd8a323b/why-signtool-return-error-0x800700c1?forum=wdk) suggests that error code indicates a incorrectly formatted DLL.
I used "CFF Explorer VIII" to open InstallerCustomActions.CA.dll. I observed that the [File Header]\[Machine] field is 0x14C/Intel386 and [Optional Header]\[Magic] field is 0x010B/PE32. These seem suspicious since I'm building a 64-bit DLL for a 64-bit installer.
This is my makesfxca call, and I've listed what CFF Explorer indicates for the Machine/Magic fields. 0x8664 maps to AMD64 and 0x020B maps to PE64 in CFF Explorer.
$(TargetDir)\InstallerCustomActions64.CA.dll ^ [0x8664 / 0x020B]
$(TargetDir)\sfxca.dll ^ [0x8664 / 0x020B]
$(TargetDir)\InstallerCustomActions.dll ^ [0x8664 / 0x020B]
$(TargetDir)\redacted.exe ^ [0x8664 / 0x020B]
$(TargetDir)\Microsoft.Deployment.WindowsInstaller.dll ^ [0x14C / 0x010B]
$(TargetDir)\redacted.dll ^ [0x8664 / 0x020B]
$(TargetDir)\redacted.sys ^ [0x8664 / 0x020B]
I observed that the Microsoft.Deployment.WindowsInstaller.dll is also set to 0x14C/0x010B, like the InstallerCustomActions.CA.dll.
This seems to suggest that a possible root-cause for signtool.exe failures might be caused by MakeSfxCA using a 32-bit version of Microsoft.Deployment.WindowsInstaller.dll. I failed to find a 64-bit version of that file in the wix toolset installation folder.
Does the wix community have a suggestion on what I should try next? The CustomAction needs to 64-bit because of the .sys file, which I cannot change.
More information about the wix-users