Subclassing and Hooking with Visual Basic

Subclassing and Hooking with Visual Basic

by Stephen Teilhet
Released June 2001
Publisher(s): O'Reilly Media, Inc.
ISBN: 9781491932681

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Subclassing & Hooking with Visual Basic offers developers a unique way to customize Windows behavior.Windows is a message-based system. Every action you request creates one or more messages to carry out the action. These messages are passed between objects and carry with them information that gives the recipient more detail on how to interpret and act upon the message.With Subclassing and the Windows hooking mechanism ("hooks"), you can manipulate, modify, or even discard messages bound for other objects within the operating system, in the process changing the way the system behaves. What kinds of results can you achieve using the power of subclassing and hooking? Here are just a few of the possibilities:

  • Determine when a window is being activated or deactivated and respond to this change.
  • Display descriptions of menu items as the mouse moves across them.
  • Disallow a user to move or resize a window.
  • Determine where the mouse cursor is and respond accordingly.
  • Determine when the display resolution has been changed.
  • Monitor the system for a low system resource condition.
  • Modify or disallow keystrokes sent to a window or a control.
  • Create an automated testing application.
  • Determine when an application is idle.
Along with this power comes responsibility; Windows is very unforgiving if subclassing and hooking are used incorrectly. Subclassing & Hooking with Visual Basic demonstrates the various techniques for intercepting messages bound for one or more windows or controls: the intercepted message can be left in its original state or modified; afterwards, the message can be sent to its original destination or discarded.For both VB 6 and VB.NET developers, Subclassing & Hooking with Visual Basic opens up a wealth of possibilities that ordinarily would be completely unavailable, or at least not easy to implement.

Publisher resources

View/Submit Errata

Table of contents

  1. Subclassing and Hooking with Visual Basic
  2. A Note Regarding Supplemental Files
  3. Preface
    1. Who This Book Is For
    2. How This Book Is Structured
    3. Obtaining the Sample Code
    4. Conventions Used in This Book
    5. How to Contact Us
    6. Acknowledgments
  4. I. Introducing Subclassing and Hooking
    1. 1. Introduction
      1. Subclassing
      2. The Window Hooking Mechanism
      3. Tools to Aid Us in Our Efforts
        1. Spy++
          1. Using Spy++ to examine a VB application
        2. NuMega SmartCheck
        3. Dbgwproc.dll
        4. Microsoft System Information
      4. A Word of Warning
    2. 2. Windows System-Specific Information
      1. Inside a Windows Application
        1. Window Relationships
        2. What Are All These Handles For?
          1. hWnd
          2. hInstance
          3. hHook
        3. Processes
        4. Threads
        5. The Internals of a Window
          1. The window class
          2. The window procedure
          3. Creating a simple window application with Visual C++
      2. Inside the Windows Messaging System
        1. The Raw Input Thread
        2. The System Message Queue
        3. Thread Message Queues
        4. Message Loops
        5. Messages
        6. The Different Types of Messages
      3. Performance Considerations
    3. 3. The Basics of Subclassing and Hooks
      1. What Is Subclassing?
        1. How Does Subclassing Work?
        2. The Types of Subclassing
          1. Instance subclassing
          2. Global subclassing
          3. Superclassing
        3. Why Do We Use Subclassing?
      2. What Are Hooks?
        1. How Do Hooks Work?
        2. Hook Scope
          1. Thread-specific
          2. System-wide
        3. Why Do We Use Hooks?
        4. Deciding Between Hooking or Subclassing
  5. II. Subclassing and Superclassing
    1. 4. Subclassing
      1. The AddressOf Operator
        1. Using AddressOf
        2. Callbacks and Enumeration Functions
        3. Callbacks and Asynchronous Processing
        4. AddressOf and Subclassing
      2. Some Subclassing Tips
      3. Instance Subclassing: An Example
        1. The Class Module
        2. The BAS Module
        3. The Form Module
        4. Doing Something Interesting
          1. Overriding window resizing
          2. Overriding a window’s minimize behavior
      4. Global Subclassing
        1. Changes to the Class
        2. Changes to the BAS Module
        3. Changes to the Form
        4. How It All Works
        5. Behind the Scenes with Spy++
      5. Summary of Key Points in Subclassing
    2. 5. Subclassing the Windows Common Dialog Boxes
      1. How Common Dialog BoxSubclassing Works
        1. Using the COMDLG32.DLL
        2. The Resource File
        3. Creating a Dialog Resource File UsingVB and Visual C++
        4. Manipulating Control Placement on the Dialog
      2. Using the Resource
        1. Embedding the Resource in the EXE
        2. Using a Resource in a DLL
          1. Creating a Visual C++ resource DLL
          2. Creating a VB resource DLL
      3. Creating the Subclassing Application
        1. The Code Behind the frmMain.frm Module
        2. The Code Behind the Module1.bas Module
          1. The dialog creation structure
          2. The code for the Module1.bas module
          3. The dialog hook procedure
          4. Dialog initialization
          5. Processing dialog box notification messages
          6. Processing messages from the controls on the child dialog box
          7. Processing the dialog shutdown messages
          8. Default message processing
      4. Subclassing Common Dialog Boxes Other Than Open and Save As
        1. The Dialog Template Resource
          1. Creating a dialog resource file
          2. The Color common dialog
          3. The Font common dialog
          4. The Print common dialog
          5. The Page Setup common dialog
          6. The Print Property Sheet common dialog
          7. The Find common dialog
          8. The Replace common dialog
        2. Receiving Notification and Control Messagesfrom the Dialog
      5. Problems Subclassing the Find and Replace Common Dialogs
    3. 6. ActiveX Controls and Subclassing
      1. Subclassing a Third-PartyActiveX Control
      2. Subclassing an ActiveX ControlCreated in VB
        1. Creating the Control
        2. Creating the Project to Subclass the Control
      3. Subclassing a UserControl from Within a VB-Created ActiveX Control
        1. Creating the Control
          1. The UserControl module
          2. The Module1.bas module
        2. Hosting the Control
      4. Creating an ActiveX Control That Subclasses Other Windows
        1. Creating the First Version of the Control
        2. Creating the Second Version of the Control
        3. Creating the Third Version of the Control
    4. 7. Superclassing
      1. What Is Superclassing?
        1. Similarities and Differences Between Subclassing and Superclassing
        2. When to Use Superclassing
      2. How the Superclassing Example Works
        1. The Code
          1. The module-level declarations
          2. The Sub Main procedure
          3. Message loop operation
          4. Class creation
          5. Window creation
          6. The superclass procedures
      3. Peering into the Superclassing Application with Spy++
    5. 8. Debugging Techniques for Subclassing
      1. Where to Start
        1. Subclassing Checklist
        2. Stepping Through the Application
        3. Log Files
      2. Microsoft Tools
        1. Spy++
        2. DBGWPROC.DLL
      3. NuMega’s SmartCheck
  6. III. Hooking
    1. 9. WH_CALLWNDPROC
      1. How WH_CALLWNDPROC Works
        1. Filter Function Definition
        2. Location of This Hook in the System
      2. A Thread-Specific Example
      3. Caveats
    2. 10. WH_CALLWNDPROCRET
      1. How WH_CALLWNDPROCRET Works
        1. Filter Function Definition
        2. Location of This Hook in the System
      2. A Thread-Specific Example
      3. Caveats
    3. 11. WH_GETMESSAGE
      1. How WH_GETMESSAGE Works
        1. Filter Function Definition
        2. Location of This Hook in the System
      2. A Thread-Specific Example
        1. Other Uses of This Hook
      3. Caveats
    4. 12. WH_KEYBOARD and WH_KEYBOARD_LL
      1. How Keyboard Messaging Operates
      2. Description of Hooks
        1. Filter Function Definitions
        2. Explanation of Hook Codes
        3. Location of This Hook in the System
      3. A Thread-Specific Example
      4. A System-Wide Example
      5. A Low-Level Hook Example
      6. Caveats
    5. 13. WH_MOUSE and WH_MOUSE_LL
      1. How Mouse Messaging Operates
        1. The Mouse Capture
        2. Mouse Click Messages
      2. Description of Hooks
        1. Filter Function Definitions
        2. Explanation of Hook Codes
      3. Location of This Hook in the System
      4. A Single-Thread Example
      5. A System-Wide Hook Example
        1. Low-Level Mouse Hook Example
      6. Caveats
    6. 14. WH_FOREGROUNDIDLE
      1. Description
      2. Location of This Hook in the System
      3. Background Processing Example
      4. Caveats
    7. 15. WH_MSGFILTER
      1. Description
        1. The Modal Loop
        2. Filter Function Definition
        3. Explanation of Hook Codes
        4. Location of This Hook in the System
      2. A Thread-Specific Example
      3. Caveats
    8. 16. WH_SYSMSGFILTER
      1. Description
        1. Filter Function Definition
        2. Explanation of Hook Codes
        3. Location of This Hook in the System
      2. A System-Wide Example
      3. Caveats
    9. 17. WH_SHELL
      1. Description
        1. Filter Function Definition
        2. Explanation of Hook Codes
      2. Location of This Hook in the System
      3. A Thread-Specific Example
        1. Using the Application
        2. The Code
      4. Caveats
    10. 18. WH_CBT
      1. Description
        1. Filter Function Definition
        2. Explanation of Hook Codes
        3. Comparing WH_SHELL to WH_CBT
      2. A Thread-Specific Example
        1. Using the WH_JOURNALPLAYBACK Hookwith a CBT
      3. Caveats
    11. 19. WH_JOURNALRECORD
      1. Description
        1. Stopping This Hook
        2. Explanation of Hook Codes
        3. Location of This Hook in the System
      2. The Macro Recorder Example
      3. Caveats
    12. 20. WH_JOURNALPLAYBACK
      1. Description
        1. Explanation of Hook Codes
        2. Location of This Hook in the System
      2. The Macro Recorder/Playback Example
      3. Caveats
    13. 21. WH_DEBUG
      1. Description
        1. Filter Function Definition
        2. Explanation of Hook Codes
      2. A Thread-Specific Example
      3. Some WH_DEBUG Scenarios
        1. Problem: Misbehaving System-Wide Hook
        2. Problem: Determining if Hooks Are Interfering with Each Other
        3. Problem: Performance Degradation
      4. Caveats
  7. IV. The .NET Platform
    1. 22. Subclassing .NET WinForms
      1. What Is a WinForm?
        1. Using a WinForm
      2. Instance Subclassing
        1. Technique #1: Overriding the WinProc Method
          1. The application
          2. The .VB file
        2. Technique #2: Binding to a NativeWindow Class Instance
          1. The application
          2. The .VB file
        3. Technique #3: The IMessageFilter Interface
          1. The application
          2. The .VB file
      3. Subclassing a Control
        1. The application
          1. The .VB file
      4. Superclassing
        1. The application
          1. The .VB file
      5. Superclassing a Control
        1. The application
          1. The .VB file
      6. Intercepting Keystrokes in a WinForm
        1. The application
          1. The .VB file
      7. Behind the Scenes with Spy++
    2. 23. Implementing Hooks in VB.NET
      1. Hooks and VB.NET
      2. Delegates
        1. A Simple Example Using Delegates
      3. Implementing the WH_KEYBOARD Hook
        1. The Application
        2. The .VB File
      4. Implementing the WH_MOUSE Hook
        1. The Application
        2. The .VB File
  8. V. Appendixes
    1. A. Messages
      1. Window Messages
        1. Public Const WM_NULL = &H0
        2. Public Const WM_CREATE = &H1
        3. Public Const WM_DESTROY = &H2
        4. Public Const WM_MOVE = &H3
        5. Public Const WM_SIZE = &H5
        6. Public Const WM_ACTIVATE = &H6
        7. Public Const WM_SETFOCUS = &H7
        8. Public Const WM_KILLFOCUS = &H8
        9. Public Const WM_SETTEXT = &HC
        10. Public Const WM_PAINT = &HF
        11. Public Const WM_QUIT = &H12
        12. Public Const WM_ERASEBKGND = &H14
        13. Public Const WM_SHOWWINDOW = &H18
        14. Public Const WM_SETCURSOR = &H20
        15. Public Const WM_QUEUESYNC = &H23
        16. Public Const WM_GETMINMAXINFO = &H24
        17. Public Const WM_WINDOWPOSCHANGING = &H46
        18. Public Const WM_WINDOWPOSCHANGED = &H47
        19. Public Const WM_CANCELJOURNAL = &H4B
        20. Public Const WM_NCCREATE = &H81
        21. Public Const WM_NCDESTROY = &H82
        22. Public Const WM_NCCALCSIZE = &H83
        23. Public Const WM_NCHITTEST = &H84
        24. Public Const WM_NCPAINT = &H85
        25. Public Const WM_NCACTIVATE = &H86
        26. Public Const WM_GETDLGCODE = &H87
        27. Public Const WM_NCMOUSEMOVE = &HA0
        28. Public Const WM_NCLBUTTONDOWN = &HA1
        29. Public Const WM_NCLBUTTONUP = &HA2
        30. Public Const WM_NCLBUTTONDBLCLK = &HA3
        31. Public Const WM_NCRBUTTONDOWN = &HA4
        32. Public Const WM_NCRBUTTONUP = &HA5
        33. Public Const WM_NCRBUTTONDBLCLK = &HA6
        34. Public Const WM_NCMBUTTONDOWN = &HA7
        35. Public Const WM_NCMBUTTONUP = &HA8
        36. Public Const WM_NCMBUTTONDBLCLK = &HA9
        37. Public Const WM_NCXBUTTONDOWN = &HAB
        38. Public Const WM_NCXBUTTONUP = &HAC
        39. Public Const WM_NCXBUTTONDBLCLK = &HAD
        40. Public Const WM_KEYDOWN = &H100
        41. Public Const WM_KEYUP = &H101
        42. Public Const WM_CHAR = &H102
        43. Public Const WM_SYSKEYDOWN = &H104
        44. Public Const WM_SYSKEYUP = &H105
        45. Public Const WM_SYSCHAR = &H106
        46. Public Const WM_INITDIALOG = &H110
        47. Public Const WM_COMMAND = &H111
        48. Public Const WM_SYSCOMMAND = &H112
        49. Public Const WM_ENTERIDLE = &H121
        50. Public Const WM_CTLCOLOREDIT = &H133
        51. Public Const WM_CTLCOLORLISTBOX = &H134
        52. Public Const WM_CTLCOLORBTN = &H135
        53. Public Const WM_CTLCOLORDLG = &H136
        54. Public Const WM_CTLCOLORSCROLLBAR = &H137
        55. Public Const WM_CTLCOLORSTATIC = &H138
        56. Public Const WM_MOUSEMOVE = &H200
        57. Public Const WM_LBUTTONDOWN = &H201
        58. Public Const WM_LBUTTONUP = &H202
        59. Public Const WM_LBUTTONDBLCLK = &H203
        60. Public Const WM_RBUTTONDOWN = &H204
        61. Public Const WM_RBUTTONUP = &H205
        62. Public Const WM_RBUTTONDBLCLK = &H206
        63. Public Const WM_MBUTTONDOWN = &H207
        64. Public Const WM_MBUTTONUP = &H208
        65. Public Const WM_MBUTTONDBLCLK = &H209
        66. Public Const WM_XBUTTONDOWN = &H20B
        67. Public Const WM_XBUTTONUP = &H20C
        68. Public Const WM_XBUTTONDBLCLK = &H20D
        69. Public Const WM_PARENTNOTIFY = &H210
        70. Public Const WM_ENTERMENULOOP = &H211
        71. Public Const WM_EXITMENULOOP = &H212
        72. Public Const WM_SIZING = &H214
        73. Public Const WM_CAPTURECHANGED = &H215
        74. Public Const WM_MOVING = &H216
        75. Public Const WM_ENTERSIZEMOVE = &H231
        76. Public Const WM_EXITSIZEMOVE = &H232
        77. Public Const WM_HOTKEY = &H312
      2. Button Control-Specific Messages
        1. Public Const BM_GETCHECK = &HF0
        2. Public Const BM_SETCHECK = &HF1
        3. Public Const BM_CLICK = &HF5
      3. Combo Box-Specific Messages
        1. Public Const CB_ADDSTRING = &H143
        2. Public Const CB_DELETESTRING = &H144
        3. Public Const CB_GETCOUNT = &H146
        4. Public Const CB_GETCURSEL = &H147
      4. Edit Control-Specific Messages
        1. Public Const EM_GETLINE = &HC4
        2. Public Const EM_CANUNDO = &HC6
        3. Public Const EM_UNDO = &HC7
      5. Listbox-Specific Messages
        1. Public Const LB_ADDSTRING = &H180
        2. Public Const LB_DELETESTRING = &H182
        3. Public Const LB_SETSEL = &H185
        4. Public Const LB_GETSEL = &H187
      6. Scroll Bar-Specific Messages
        1. Public Const SBM_SETPOS = &HE0
        2. Public Const SBM_GETPOS = &HE1
        3. Public Const SBM_SETRANGE = &HE2
        4. Public Const SBM_GETRANGE = &HE3
      7. Messages Specific to the CommonDialog Boxes
        1. Public Const CDM_GETFILEPATH = (CDM_FIRST + &H1)
        2. Public Const CDM_GETFOLDERPATH = (CDM_FIRST + &H2)
        3. Public Const CDM_GETSPEC = (CDM_FIRST + &H0)
        4. Public Const CDM_HIDECONTROL = (CDM_FIRST + &H5)
        5. Public Const CDM_SETCONTROLTEXT = (CDM_FIRST + &H4)
        6. Public Const CDM_SETDEFEXT = (CDM_FIRST + &H6)
      8. Messages Specific to the Font Common Dialog Box
        1. Public Const WM_CHOOSEFONT_GETLOGFONT = _USER + 1)
        2. Public Const WM_CHOOSEFONT_SETLOGFONT = (WM_USER + 101)
        3. Public Const WM_CHOOSEFONT_SETFLAGS = (WM_USER + 102)
      9. Messages Specific to the Page Setup Common Dialog Box
        1. Public Const WM_PSD_PAGESETUPDLG = (WM_USER)
        2. Public Const WM_PSD_FULLPAGERECT = (WM_USER+1)
        3. Public Const WM_PSD_MINMARGINRECT = (WM_USER+2)
        4. Public Const WM_PSD_MARGINRECT = (WM_USER+3)
        5. Public Const WM_PSD_ENVSTAMPRECT = (WM_USER+5)
        6. Public Const WM_PSD_FULLPAGERECT = (WM_USER+6)
    2. B. API Functions
      1. AttachThreadInput
      2. BroadcastSystemMessage
      3. CallNextHookEx
      4. CallWindowProc
      5. ChooseColor
      6. ChooseFont
      7. CopyMemory
      8. CreateWindowEx
      9. DefWindowProc
      10. EnumChildWindows
      11. FindText
      12. FindWindow
      13. FindWindowEx
      14. GetClassInfoEx
      15. GetDeskTopWindow
      16. GetDlgItem
      17. GetMessage
      18. GetParent
      19. GetSaveFileName
      20. GetWindow
      21. GetWindowLong
      22. GetWindowRect
      23. OutputDebugStringA
      24. PageSetupDlg
      25. PeekMessage
      26. PostMessage
      27. PostQuitMessage
      28. PostThreadMessage
      29. PrintDlg
      30. PrintDlgEx
      31. RegisterClassEx
      32. RegisterWindowMessage
      33. ReleaseCapture
      34. ReplaceText
      35. SendDlgItemMessage
      36. SendMessage
      37. SendMessageCallback
      38. SendMessageTimeout
      39. SendNotifyMessage
      40. SetCapture
      41. SetClassLongPtr
      42. SetWindowLongPtr
      43. SetWindowPos
      44. SetWindowsHookEx
      45. ShowWindow
      46. UnhookWindowsHookEx
      47. UnregisterClass
    3. C. Structures and Constants
      1. CBT_CREATEWND
        1. CBT_CREATEWND
      2. CBTACTIVATESTRUCT
        1. CBTACTIVATESTRUCT
      3. CHOOSECOLOR
        1. CHOOSECOLOR
      4. CHOOSEFONT
        1. CHOOSEFONT
      5. CWPRETSTRUCT
        1. CWPRETSTRUCT
      6. CWPSTRUCT
        1. CWPSTRUCT
      7. DEBUGHOOKINFO
        1. DEBUGHOOKINFO
      8. EVENTMSG
        1. EVENTMSG
      9. FINDREPLACE
        1. FINDREPLACE
      10. KBDLLHOOKSTRUCT
        1. KBDLLHOOKSTRUCT
      11. MOUSEHOOKSTRUCT
        1. MOUSEHOOKSTRUCT
      12. MOUSEHOOKSTRUCTEX
        1. MOUSEHOOKSTRUCTEX
      13. MSG
        1. MSG
      14. MSLLHOOKSTRUCT
        1. MSLLHOOKSTRUCT
      15. NMHDR
        1. NMHDR
      16. OPENFILENAME
        1. OPENFILENAME
      17. PAGESETUPDLG
        1. PAGESETUPDLG
      18. POINTAPI
        1. POINTAPI
      19. PRINTDLG
        1. PRINTDLG
      20. PRINTDLGEX
        1. PRINTDLGEX
      21. RECT
        1. RECT
      22. WNDCLASSEX
        1. WNDCLASSEX
      23. Message (.NET Structure)
        1. Message (.NET Structure)
      24. Constants
        1. BroadcastSystemMessage Function Masks
        2. Common Dialog Box Error Messages
        3. Common Dialog Box Notification Messages
        4. Class Styles
        5. Edit Control Notification Codes
        6. Extended Window Styles
        7. GetClassLongPtr Function Constants
        8. GetWindow API Function Constants
        9. GetWindowLongPtr Function Constants
        10. Hook Codes
          1. General
          2. Computer-based training (CBT) hook codes
          3. WH_MSGFILTER hook codes
          4. WH_SHELL hook codes
        11. HSHELL_ACCESSIBILITYSTATE Constants
        12. Key State Masks for Mouse Messages
        13. Low-Level Hook Constants
        14. Mouse Wheel Constant
        15. Open and Save As Common Dialog BoxNotification Codes
        16. PeekMessage Function Options
        17. PostMessage and SendMessage Function Constants
        18. Predefined Control ID Values for the Open andSave As Common Dialog Boxes
        19. SendMessageTimeout API Function Constants
        20. SetWindowLongPtr/GetWindowLongPtr Function Constants
        21. SetWindowPos Function Constants
        22. ShowWindow Function Constants
        23. User Button Notification Codes
        24. Virtual Key Codes
        25. Window Hook Constants
        26. Window Styles
        27. WM_KEYUP, WM_DOWN, andWM_CHAR lParam HIWORD Flags
        28. WM_NCHITTEST Codes
        29. WH_SHELL Codes for the HSHELL_APPCOMMAND Hook Code
        30. WM_SIZE Message wParam Values
        31. WM_SYSCOMMAND Message wParam Constants
        32. XButton Constants
  9. Index
  10. About the Author
  11. Colophon
  12. Copyright

Product information

  • Title: Subclassing and Hooking with Visual Basic
  • Author(s): Stephen Teilhet
  • Release date: June 2001
  • Publisher(s): O'Reilly Media, Inc.
  • ISBN: 9781491932681