[wix-devs] Discussion on fix Issue SqlExtension Incompatible with TLS 1.2 #5543

[Rajeev Bansal (WSSC)]

I posted this yesterday, but seems blocked due to size.
Trying without attachment.

Thanks,
Rajeev

From: Rajeev Bansal (WSSC)
Sent: Wednesday, March 7, 2018 9:48 AM
To: 'wix-devs at lists.wixtoolset.org' <wix-devs at lists.wixtoolset.org>
Subject: Discussion on fix Issue SqlExtension Incompatible with TLS 1.2 #5543

Hi,

For the issue: https://github.com/wixtoolset/issues/issues/5543   SqlExtension Incompatible with TLS 1.2 #5543
Here is the overview of how I am planning to fix it.

Root Cause and Fix
SQLOLEDB does not have TLS1.2 support, so need to use SQL native Client for it.


In the SqlConnectDatabase() API in file ..\wix3\src\libs\dutil\sqlutil.cpp

CoCreateInstance is being called with CLSID_SQLOLEDB as the first argument.
                                hr = ::CoCreateInstance(CLSID_SQLOLEDB, NULL, CLSCTX_INPROC_SERVER,  IID_IDBInitialize, (LPVOID*)&pidbInitialize);

Fix would be to call it with SQLNCLI_CLSID and if that fails, then as a fallback call with CLSID_SQLOLEDB.

OLD CODE:
                //obtain access to the SQLOLEDB provider
                hr = ::CoCreateInstance(SQLNCLI_CLSID, NULL, CLSCTX_INPROC_SERVER,
                                IID_IDBInitialize, (LPVOID*)&pidbInitialize);
                ExitOnFailure(hr, "failed to create IID_IDBInitialize object");

NEW CODE:
                //obtain access to the SQLOLEDB provider
                hr = ::CoCreateInstance(SQLNCLI_CLSID, NULL, CLSCTX_INPROC_SERVER,
                                IID_IDBInitialize, (LPVOID*)&pidbInitialize);

                if (FAILED(hr))
                {
                hr = ::CoCreateInstance(CLSID_SQLOLEDB, NULL, CLSCTX_INPROC_SERVER,
                                                IID_IDBInitialize, (LPVOID*)&pidbInitialize);
                }
                ExitOnFailure(hr, "failed to create IID_IDBInitialize object");


Other related changes:
1. SQLNCLI.h file has to be copied to ...\wix3\src\libs\dutil\inc folder
2. SQLNCLI.h has to be included in the ....\wix3\src\libs\dutil\dutil.vcxproj
3. SQLNCLI.h has to be included in the ....\wix3\src\ext\ca\serverca\scasched\precomp.h
4. SQLNCLI.h has to be included in the ....\wix3\src\libs\dutil\inc\sqlutil.h
5. #define _SQLNCLI_OLEDB_IGNORE_DEPRECATION_WARNING_ needs to be defined in file ..\wix3\src\libs\dutil\sqlutil.cpp

                Attached is the header file that needs to be added to project.

Please let me know your thoughts on it.

Thanks,
Rajeev