[wix-devs] Strange RichEdit behavior in Burn with Turkish keyboard

[Denis Z.]

Hi all!

As was reported by our support team of the product which I maintain
Burn-based installer for, there's a strange issue (fix is ready - see
below) when installer is run on a system with Turkish language installed.
This is reproduced easily (at least on Windows 7, but the effect is most
obvious on Windows 10):

0. Prerequisites: Have some Burn based installer, configured to use
standard WixStdBA module with RtfLargeLicense theme. For example,
vcredist_x86.exe from MSVC2015 will do.
1. Install Turkish language in the system, along with Turkish keyboard
layout (e.g. TRQ)
2. Open CMD console, switch to TRQ layout (or just run from Desktop with
TRQ layout active)
3. Run the installer you prepared
4. Quit installer, either cancel or let it run to the end
5. Notice the active keyboard layout, on step 3 and step 4

Expected result: Active keyboard layout doesn't change (so it still should
be TRQ on step 3 and 4).
Actual result: Active keyboard layout changes to the previous one before
TRQ was set (e.g. ENG).

The strange thing is this is only happening with Turkish layouts. Spanish,
German, Russian, etc. does not show such behavior with Burn.

As I was able to investigate, the bug concluded in RichEdit control which
is used in RtfLargeLicense theme. For some reason, the code inside
RICHED20.DLL calls ActivateKeyboardLayout with different layout than
currently selected. Previously GetKeyState with VK_SHIFT argument reports
incorrect state for Shift key(s), so may be this triggers the bug (this
doesn't happen with other layouts).

So while all this doesn't relate to Burn code, there's an easy fix possible
- just use MSFTEDIT_CLASS when you create RichEdit control (the newer one),
instead of RICHEDIT_CLASS which is used currently (old code). This change
also implies changing the library to load - now it's MSFTEDIT.DLL, instead
of RICHED20.DLL. This library exists at least on Windows XP SP3 (cannot
check older SP) and later versions, and works perfectly as a drop-in
replacement. Probably other issues could be fixed as well, if any (on
Win10, this is of version 3.1 vs. version 8.5 of MSFTEDIT).

On order to deploy this fix, it is only necessary to change a couple of
lines in thmutil.cpp (around "case THEME_CONTROL_TYPE_RICHEDIT:") and
PrintEula.cpp (similar lines, for MSI-based UI) and rebuild WixStdBA.dll.
Now replace <BootstrapperApplicationRef> element with
<BootstrapperApplication>, add (now missing) <PayloadGroupRef
Id='WixStdbaRtfLargeLicensePayloads' /> to the content, as well as existing
<bal:WixStandardBootstrapperApplication/> and you are done.

Hope this fix will be included in later WiX builds.

--
Best regards,
Denis.