Debugging DLL Custom Actions in InstallShield Professional - Windows Installer Edition 2.x

The current release of IPWI (2.0) provides a feature to ease your work to debug a C/C++ based user custom DLL function. This is an undocumented feature. You need to follow these steps to use it:

Build a debug version of your C/C++ custom action DLL.
Add a new custom action to your project which calls a function from your debug version of the DLL, the custom action has to be of type "Call a function in a standard dynamic-link library". (For functions that only take the MSI database Handle as the parameter, which is usually considered as "Call a function in a Windows Installer dynamic-link library", you can still make them of type "Call a function in a standard dynamic-link library", in the argument list window, pick up following selections from the drop down combo box: type as "HANDLE", source as "Constant" and value as "MsiHandle").
Make sure you enable "Just-in-time debugging" of your debugger. (For VC, goto Tools -> Options-> Debugger). After this, you may want to make sure your debugger is registered:

For WinNT/Win2k, check the following value in the registry:
"HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug\Debugger"

For Windows98, check the key "Debugger" under section "AeDebug" of win.ini file.

You should see your debugger there. If not, manually modify this key so your favorite debugger will be used.
Create the following key in the registry:
HKEY_LOCAL_MACHINE\SOFTWARE\InstallShield\ISWI\2.0\DebugDLLCustomAction (key, not value).

Now you are ready for the debug session, start your package (Either click "run" button or "test" button from IPWI IDE). Before calling your DLL function, setup will prompt a message box asking whether you want to debug the given function. Say no if you are not interested. If you say yes, your debugger will be launched and your installation will break inside the debugger. (You may get another message box saying a breakpoint is reached, depending on your debugger) You will find your debugger breaks at the following assembly code (It may look a little different on your machine)

1001748E int 3
1001748F call dword ptr [ebp-0Ch]

The second line is the call to your function. Make three single steps until you are inside your own function. Most probably you will see a mix of C/C++ code and assembly code. Do not be scared, just turn off the assembly mode (With Visual C++, Press Ctrl+F11) Now you should see the debugger stops at the first line of your code. You can continue your session from there.

This works for custom actions in both UI and Execute sequences.

Remove the registry key mentioned above to disable this feature.

Currently, this has been tested on Windows 98 and Windows NT/2000 with Microsoft Visual Studio 6. It may not work on Windows 95.

Please send your feedback to mingbiaof@installshield.com

	News	Discussions	Windows Installer	Related Tools	More Help	InstallScript	About InstallSite	Shop	Site Search
	Neuigkeiten	Diskussionsgruppen	Windows Installer	MSI FAQ	Artikel			Shop	Suche

Copyright © by InstallSite Stefan Krueger. All rights reserved. Legal information.
Impressum/Imprint Datenschutzerklärung/Privacy Policy
By using this site you agree to the license agreement. Webmaster contact