Subclassing and Hooking with Visual Basic

Subclassing and Hooking with Visual Basic

by Stephen Teilhet
Released June 2001
Publisher(s): O'Reilly Media, Inc.
ISBN: 9780596001186

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Buy on Amazon Buy on ebooks.com
Start your free trial

Book description

Subclassing & Hooking with Visual Basic offersdevelopers a unique way to customize Windows behavior. Windows is amessage-based system. Every action you request creates one or moremessages to carry out the action. These messages are passed betweenobjects and carry with them information that gives the recipientmore detail on how to interpret and act upon the message. WithSubclassing and the Windows hooking mechanism ("hooks"), you canmanipulate, modify, or even discard messages bound for otherobjects within the operating system, in the process changing theway the system behaves. What kinds of results can you achieve usingthe power of subclassing and hooking? Here are just a few of thepossibilities:

  • Determine when a window is being activated ordeactivated and respond to this change.

  • Display descriptions of menu items as the mousemoves across them.

  • Disallow a user to move or resize a window.

  • Determine where the mouse cursor is and respondaccordingly.

  • Determine when the display resolution has beenchanged.

  • Monitor the system for a low system resourcecondition.

  • Modify or disallow keystrokes sent to a window or acontrol.

  • Create an automated testing application.

  • Determine when an application is idle.

    • Along with this power comes responsibility; Windows is veryunforgiving if subclassing and hooking are used incorrectly.Subclassing & Hooking with Visual Basic demonstrates thevarious techniques for intercepting messages bound for one or morewindows or controls: the intercepted message can be left in itsoriginal state or modified; afterwards, the message can be sent toits original destination or discarded. For both VB 6 and VB.NETdevelopers, Subclassing & Hooking with Visual Basicopens up a wealth of possibilities that ordinarily would becompletely unavailable, or at least not easy to implement.

    Table of contents

    1. Subclassing and Hooking with Visual Basic
    2. A Note Regarding Supplemental Files
    3. Preface
      1. Who This Book Is For
      2. How This Book Is Structured
      3. Obtaining the Sample Code
      4. Conventions Used in This Book
      5. How to Contact Us
      6. Acknowledgments
    4. I. Introducing Subclassing and Hooking
      1. 1. Introduction
        1. Subclassing
        2. The Window Hooking Mechanism
        3. Tools to Aid Us in Our Efforts
          1. Spy++
            1. Using Spy++ to examine a VB application
          2. NuMega SmartCheck
          3. Dbgwproc.dll
          4. Microsoft System Information
        4. A Word of Warning
      2. 2. Windows System-Specific Information
        1. Inside a Windows Application
          1. Window Relationships
          2. What Are All These Handles For?
            1. hWnd
            2. hInstance
            3. hHook
          3. Processes
          4. Threads
          5. The Internals of a Window
            1. The window class
            2. The window procedure
            3. Creating a simple window application with Visual C++
        2. Inside the Windows Messaging System
          1. The Raw Input Thread
          2. The System Message Queue
          3. Thread Message Queues
          4. Message Loops
          5. Messages
          6. The Different Types of Messages
        3. Performance Considerations
      3. 3. The Basics of Subclassing and Hooks
        1. What Is Subclassing?
          1. How Does Subclassing Work?
          2. The Types of Subclassing
            1. Instance subclassing
            2. Global subclassing
            3. Superclassing
          3. Why Do We Use Subclassing?
        2. What Are Hooks?
          1. How Do Hooks Work?
          2. Hook Scope
            1. Thread-specific
            2. System-wide
          3. Why Do We Use Hooks?
          4. Deciding Between Hooking or Subclassing
    5. II. Subclassing and Superclassing
      1. 4. Subclassing
        1. The AddressOf Operator
          1. Using AddressOf
          2. Callbacks and Enumeration Functions
          3. Callbacks and Asynchronous Processing
          4. AddressOf and Subclassing
        2. Some Subclassing Tips
        3. Instance Subclassing: An Example
          1. The Class Module
          2. The BAS Module
          3. The Form Module
          4. Doing Something Interesting
            1. Overriding window resizing
            2. Overriding a window’s minimize behavior
        4. Global Subclassing
          1. Changes to the Class
          2. Changes to the BAS Module
          3. Changes to the Form
          4. How It All Works
          5. Behind the Scenes with Spy++
        5. Summary of Key Points in Subclassing
      2. 5. Subclassing the Windows Common Dialog Boxes
        1. How Common Dialog BoxSubclassing Works
          1. Using the COMDLG32.DLL
          2. The Resource File
          3. Creating a Dialog Resource File UsingVB and Visual C++
          4. Manipulating Control Placement on the Dialog
        2. Using the Resource
          1. Embedding the Resource in the EXE
          2. Using a Resource in a DLL
            1. Creating a Visual C++ resource DLL
            2. Creating a VB resource DLL
        3. Creating the Subclassing Application
          1. The Code Behind the frmMain.frm Module
          2. The Code Behind the Module1.bas Module
            1. The dialog creation structure
            2. The code for the Module1.bas module
            3. The dialog hook procedure
            4. Dialog initialization
            5. Processing dialog box notification messages
            6. Processing messages from the controls on the child dialog box
            7. Processing the dialog shutdown messages
            8. Default message processing
        4. Subclassing Common Dialog Boxes Other Than Open and Save As
          1. The Dialog Template Resource
            1. Creating a dialog resource file
            2. The Color common dialog
            3. The Font common dialog
            4. The Print common dialog
            5. The Page Setup common dialog
            6. The Print Property Sheet common dialog
            7. The Find common dialog
            8. The Replace common dialog
          2. Receiving Notification and Control Messagesfrom the Dialog
        5. Problems Subclassing the Find and Replace Common Dialogs
      3. 6. ActiveX Controls and Subclassing
        1. Subclassing a Third-PartyActiveX Control
        2. Subclassing an ActiveX ControlCreated in VB
          1. Creating the Control
          2. Creating the Project to Subclass the Control
        3. Subclassing a UserControl from Within a VB-Created ActiveX Control
          1. Creating the Control
            1. The UserControl module
            2. The Module1.bas module
          2. Hosting the Control
        4. Creating an ActiveX Control That Subclasses Other Windows
          1. Creating the First Version of the Control
          2. Creating the Second Version of the Control
          3. Creating the Third Version of the Control
      4. 7. Superclassing
        1. What Is Superclassing?
          1. Similarities and Differences Between Subclassing and Superclassing
          2. When to Use Superclassing
        2. How the Superclassing Example Works
          1. The Code
            1. The module-level declarations
            2. The Sub Main procedure
            3. Message loop operation
            4. Class creation
            5. Window creation
            6. The superclass procedures
        3. Peering into the Superclassing Application with Spy++
      5. 8. Debugging Techniques for Subclassing
        1. Where to Start
          1. Subclassing Checklist
          2. Stepping Through the Application
          3. Log Files
        2. Microsoft Tools
          1. Spy++
          2. DBGWPROC.DLL
        3. NuMega’s SmartCheck
    6. III. Hooking
      1. 9. WH_CALLWNDPROC
        1. How WH_CALLWNDPROC Works
          1. Filter Function Definition
          2. Location of This Hook in the System
        2. A Thread-Specific Example
        3. Caveats
      2. 10. WH_CALLWNDPROCRET
        1. How WH_CALLWNDPROCRET Works
          1. Filter Function Definition
          2. Location of This Hook in the System
        2. A Thread-Specific Example
        3. Caveats
      3. 11. WH_GETMESSAGE
        1. How WH_GETMESSAGE Works
          1. Filter Function Definition
          2. Location of This Hook in the System
        2. A Thread-Specific Example
          1. Other Uses of This Hook
        3. Caveats
      4. 12. WH_KEYBOARD and WH_KEYBOARD_LL
        1. How Keyboard Messaging Operates
        2. Description of Hooks
          1. Filter Function Definitions
          2. Explanation of Hook Codes
          3. Location of This Hook in the System
        3. A Thread-Specific Example
        4. A System-Wide Example
        5. A Low-Level Hook Example
        6. Caveats
      5. 13. WH_MOUSE and WH_MOUSE_LL
        1. How Mouse Messaging Operates
          1. The Mouse Capture
          2. Mouse Click Messages
        2. Description of Hooks
          1. Filter Function Definitions
          2. Explanation of Hook Codes
        3. Location of This Hook in the System
        4. A Single-Thread Example
        5. A System-Wide Hook Example
          1. Low-Level Mouse Hook Example
        6. Caveats
      6. 14. WH_FOREGROUNDIDLE
        1. Description
        2. Location of This Hook in the System
        3. Background Processing Example
        4. Caveats
      7. 15. WH_MSGFILTER
        1. Description
          1. The Modal Loop
          2. Filter Function Definition
          3. Explanation of Hook Codes
          4. Location of This Hook in the System
        2. A Thread-Specific Example
        3. Caveats
      8. 16. WH_SYSMSGFILTER
        1. Description
          1. Filter Function Definition
          2. Explanation of Hook Codes
          3. Location of This Hook in the System
        2. A System-Wide Example
        3. Caveats
      9. 17. WH_SHELL
        1. Description
          1. Filter Function Definition
          2. Explanation of Hook Codes
        2. Location of This Hook in the System
        3. A Thread-Specific Example
          1. Using the Application
          2. The Code
        4. Caveats
      10. 18. WH_CBT
        1. Description
          1. Filter Function Definition
          2. Explanation of Hook Codes
          3. Comparing WH_SHELL to WH_CBT
        2. A Thread-Specific Example
          1. Using the WH_JOURNALPLAYBACK Hookwith a CBT
        3. Caveats
      11. 19. WH_JOURNALRECORD
        1. Description
          1. Stopping This Hook
          2. Explanation of Hook Codes
          3. Location of This Hook in the System
        2. The Macro Recorder Example
        3. Caveats
      12. 20. WH_JOURNALPLAYBACK
        1. Description
          1. Explanation of Hook Codes
          2. Location of This Hook in the System
        2. The Macro Recorder/Playback Example
        3. Caveats
      13. 21. WH_DEBUG
        1. Description
          1. Filter Function Definition
          2. Explanation of Hook Codes
        2. A Thread-Specific Example
        3. Some WH_DEBUG Scenarios
          1. Problem: Misbehaving System-Wide Hook
          2. Problem: Determining if Hooks Are Interfering with Each Other
          3. Problem: Performance Degradation
        4. Caveats
    7. IV. The .NET Platform
      1. 22. Subclassing .NET WinForms
        1. What Is a WinForm?
          1. Using a WinForm
        2. Instance Subclassing
          1. Technique #1: Overriding the WinProc Method
            1. The application
            2. The .VB file
          2. Technique #2: Binding to a NativeWindow Class Instance
            1. The application
            2. The .VB file
          3. Technique #3: The IMessageFilter Interface
            1. The application
            2. The .VB file
        3. Subclassing a Control
          1. The application
            1. The .VB file
        4. Superclassing
          1. The application
            1. The .VB file
        5. Superclassing a Control
          1. The application
            1. The .VB file
        6. Intercepting Keystrokes in a WinForm
          1. The application
            1. The .VB file
        7. Behind the Scenes with Spy++
      2. 23. Implementing Hooks in VB.NET
        1. Hooks and VB.NET
        2. Delegates
          1. A Simple Example Using Delegates
        3. Implementing the WH_KEYBOARD Hook
          1. The Application
          2. The .VB File
        4. Implementing the WH_MOUSE Hook
          1. The Application
          2. The .VB File
    8. V. Appendixes
      1. A. Messages
        1. Window Messages
          1. Public Const WM_NULL = &H0
          2. Public Const WM_CREATE = &H1
          3. Public Const WM_DESTROY = &H2
          4. Public Const WM_MOVE = &H3
          5. Public Const WM_SIZE = &H5
          6. Public Const WM_ACTIVATE = &H6
          7. Public Const WM_SETFOCUS = &H7
          8. Public Const WM_KILLFOCUS = &H8
          9. Public Const WM_SETTEXT = &HC
          10. Public Const WM_PAINT = &HF
          11. Public Const WM_QUIT = &H12
          12. Public Const WM_ERASEBKGND = &H14
          13. Public Const WM_SHOWWINDOW = &H18
          14. Public Const WM_SETCURSOR = &H20
          15. Public Const WM_QUEUESYNC = &H23
          16. Public Const WM_GETMINMAXINFO = &H24
          17. Public Const WM_WINDOWPOSCHANGING = &H46
          18. Public Const WM_WINDOWPOSCHANGED = &H47
          19. Public Const WM_CANCELJOURNAL = &H4B
          20. Public Const WM_NCCREATE = &H81
          21. Public Const WM_NCDESTROY = &H82
          22. Public Const WM_NCCALCSIZE = &H83
          23. Public Const WM_NCHITTEST = &H84
          24. Public Const WM_NCPAINT = &H85
          25. Public Const WM_NCACTIVATE = &H86
          26. Public Const WM_GETDLGCODE = &H87
          27. Public Const WM_NCMOUSEMOVE = &HA0
          28. Public Const WM_NCLBUTTONDOWN = &HA1
          29. Public Const WM_NCLBUTTONUP = &HA2
          30. Public Const WM_NCLBUTTONDBLCLK = &HA3
          31. Public Const WM_NCRBUTTONDOWN = &HA4
          32. Public Const WM_NCRBUTTONUP = &HA5
          33. Public Const WM_NCRBUTTONDBLCLK = &HA6
          34. Public Const WM_NCMBUTTONDOWN = &HA7
          35. Public Const WM_NCMBUTTONUP = &HA8
          36. Public Const WM_NCMBUTTONDBLCLK = &HA9
          37. Public Const WM_NCXBUTTONDOWN = &HAB
          38. Public Const WM_NCXBUTTONUP = &HAC
          39. Public Const WM_NCXBUTTONDBLCLK = &HAD
          40. Public Const WM_KEYDOWN = &H100
          41. Public Const WM_KEYUP = &H101
          42. Public Const WM_CHAR = &H102
          43. Public Const WM_SYSKEYDOWN = &H104
          44. Public Const WM_SYSKEYUP = &H105
          45. Public Const WM_SYSCHAR = &H106
          46. Public Const WM_INITDIALOG = &H110
          47. Public Const WM_COMMAND = &H111
          48. Public Const WM_SYSCOMMAND = &H112
          49. Public Const WM_ENTERIDLE = &H121
          50. Public Const WM_CTLCOLOREDIT = &H133
          51. Public Const WM_CTLCOLORLISTBOX = &H134
          52. Public Const WM_CTLCOLORBTN = &H135
          53. Public Const WM_CTLCOLORDLG = &H136
          54. Public Const WM_CTLCOLORSCROLLBAR = &H137
          55. Public Const WM_CTLCOLORSTATIC = &H138
          56. Public Const WM_MOUSEMOVE = &H200
          57. Public Const WM_LBUTTONDOWN = &H201
          58. Public Const WM_LBUTTONUP = &H202
          59. Public Const WM_LBUTTONDBLCLK = &H203
          60. Public Const WM_RBUTTONDOWN = &H204
          61. Public Const WM_RBUTTONUP = &H205
          62. Public Const WM_RBUTTONDBLCLK = &H206
          63. Public Const WM_MBUTTONDOWN = &H207
          64. Public Const WM_MBUTTONUP = &H208
          65. Public Const WM_MBUTTONDBLCLK = &H209
          66. Public Const WM_XBUTTONDOWN = &H20B
          67. Public Const WM_XBUTTONUP = &H20C
          68. Public Const WM_XBUTTONDBLCLK = &H20D
          69. Public Const WM_PARENTNOTIFY = &H210
          70. Public Const WM_ENTERMENULOOP = &H211
          71. Public Const WM_EXITMENULOOP = &H212
          72. Public Const WM_SIZING = &H214
          73. Public Const WM_CAPTURECHANGED = &H215
          74. Public Const WM_MOVING = &H216
          75. Public Const WM_ENTERSIZEMOVE = &H231
          76. Public Const WM_EXITSIZEMOVE = &H232
          77. Public Const WM_HOTKEY = &H312
        2. Button Control-Specific Messages
          1. Public Const BM_GETCHECK = &HF0
          2. Public Const BM_SETCHECK = &HF1
          3. Public Const BM_CLICK = &HF5
        3. Combo Box-Specific Messages
          1. Public Const CB_ADDSTRING = &H143
          2. Public Const CB_DELETESTRING = &H144
          3. Public Const CB_GETCOUNT = &H146
          4. Public Const CB_GETCURSEL = &H147
        4. Edit Control-Specific Messages
          1. Public Const EM_GETLINE = &HC4
          2. Public Const EM_CANUNDO = &HC6
          3. Public Const EM_UNDO = &HC7
        5. Listbox-Specific Messages
          1. Public Const LB_ADDSTRING = &H180
          2. Public Const LB_DELETESTRING = &H182
          3. Public Const LB_SETSEL = &H185
          4. Public Const LB_GETSEL = &H187
        6. Scroll Bar-Specific Messages
          1. Public Const SBM_SETPOS = &HE0
          2. Public Const SBM_GETPOS = &HE1
          3. Public Const SBM_SETRANGE = &HE2
          4. Public Const SBM_GETRANGE = &HE3
        7. Messages Specific to the CommonDialog Boxes
          1. Public Const CDM_GETFILEPATH = (CDM_FIRST + &H1)
          2. Public Const CDM_GETFOLDERPATH = (CDM_FIRST + &H2)
          3. Public Const CDM_GETSPEC = (CDM_FIRST + &H0)
          4. Public Const CDM_HIDECONTROL = (CDM_FIRST + &H5)
          5. Public Const CDM_SETCONTROLTEXT = (CDM_FIRST + &H4)
          6. Public Const CDM_SETDEFEXT = (CDM_FIRST + &H6)
        8. Messages Specific to the Font Common Dialog Box
          1. Public Const WM_CHOOSEFONT_GETLOGFONT = _USER + 1)
          2. Public Const WM_CHOOSEFONT_SETLOGFONT = (WM_USER + 101)
          3. Public Const WM_CHOOSEFONT_SETFLAGS = (WM_USER + 102)
        9. Messages Specific to the Page Setup Common Dialog Box
          1. Public Const WM_PSD_PAGESETUPDLG = (WM_USER)
          2. Public Const WM_PSD_FULLPAGERECT = (WM_USER+1)
          3. Public Const WM_PSD_MINMARGINRECT = (WM_USER+2)
          4. Public Const WM_PSD_MARGINRECT = (WM_USER+3)
          5. Public Const WM_PSD_ENVSTAMPRECT = (WM_USER+5)
          6. Public Const WM_PSD_FULLPAGERECT = (WM_USER+6)
      2. B. API Functions
        1. AttachThreadInput
        2. BroadcastSystemMessage
        3. CallNextHookEx
        4. CallWindowProc
        5. ChooseColor
        6. ChooseFont
        7. CopyMemory
        8. CreateWindowEx
        9. DefWindowProc
        10. EnumChildWindows
        11. FindText
        12. FindWindow
        13. FindWindowEx
        14. GetClassInfoEx
        15. GetDeskTopWindow
        16. GetDlgItem
        17. GetMessage
        18. GetParent
        19. GetSaveFileName
        20. GetWindow
        21. GetWindowLong
        22. GetWindowRect
        23. OutputDebugStringA
        24. PageSetupDlg
        25. PeekMessage
        26. PostMessage
        27. PostQuitMessage
        28. PostThreadMessage
        29. PrintDlg
        30. PrintDlgEx
        31. RegisterClassEx
        32. RegisterWindowMessage
        33. ReleaseCapture
        34. ReplaceText
        35. SendDlgItemMessage
        36. SendMessage
        37. SendMessageCallback
        38. SendMessageTimeout
        39. SendNotifyMessage
        40. SetCapture
        41. SetClassLongPtr
        42. SetWindowLongPtr
        43. SetWindowPos
        44. SetWindowsHookEx
        45. ShowWindow
        46. UnhookWindowsHookEx
        47. UnregisterClass
      3. C. Structures and Constants
        1. CBT_CREATEWND
          1. CBT_CREATEWND
        2. CBTACTIVATESTRUCT
          1. CBTACTIVATESTRUCT
        3. CHOOSECOLOR
          1. CHOOSECOLOR
        4. CHOOSEFONT
          1. CHOOSEFONT
        5. CWPRETSTRUCT
          1. CWPRETSTRUCT
        6. CWPSTRUCT
          1. CWPSTRUCT
        7. DEBUGHOOKINFO
          1. DEBUGHOOKINFO
        8. EVENTMSG
          1. EVENTMSG
        9. FINDREPLACE
          1. FINDREPLACE
        10. KBDLLHOOKSTRUCT
          1. KBDLLHOOKSTRUCT
        11. MOUSEHOOKSTRUCT
          1. MOUSEHOOKSTRUCT
        12. MOUSEHOOKSTRUCTEX
          1. MOUSEHOOKSTRUCTEX
        13. MSG
          1. MSG
        14. MSLLHOOKSTRUCT
          1. MSLLHOOKSTRUCT
        15. NMHDR
          1. NMHDR
        16. OPENFILENAME
          1. OPENFILENAME
        17. PAGESETUPDLG
          1. PAGESETUPDLG
        18. POINTAPI
          1. POINTAPI
        19. PRINTDLG
          1. PRINTDLG
        20. PRINTDLGEX
          1. PRINTDLGEX
        21. RECT
          1. RECT
        22. WNDCLASSEX
          1. WNDCLASSEX
        23. Message (.NET Structure)
          1. Message (.NET Structure)
        24. Constants
          1. BroadcastSystemMessage Function Masks
          2. Common Dialog Box Error Messages
          3. Common Dialog Box Notification Messages
          4. Class Styles
          5. Edit Control Notification Codes
          6. Extended Window Styles
          7. GetClassLongPtr Function Constants
          8. GetWindow API Function Constants
          9. GetWindowLongPtr Function Constants
          10. Hook Codes
            1. General
            2. Computer-based training (CBT) hook codes
            3. WH_MSGFILTER hook codes
            4. WH_SHELL hook codes
          11. HSHELL_ACCESSIBILITYSTATE Constants
          12. Key State Masks for Mouse Messages
          13. Low-Level Hook Constants
          14. Mouse Wheel Constant
          15. Open and Save As Common Dialog BoxNotification Codes
          16. PeekMessage Function Options
          17. PostMessage and SendMessage Function Constants
          18. Predefined Control ID Values for the Open andSave As Common Dialog Boxes
          19. SendMessageTimeout API Function Constants
          20. SetWindowLongPtr/GetWindowLongPtr Function Constants
          21. SetWindowPos Function Constants
          22. ShowWindow Function Constants
          23. User Button Notification Codes
          24. Virtual Key Codes
          25. Window Hook Constants
          26. Window Styles
          27. WM_KEYUP, WM_DOWN, andWM_CHAR lParam HIWORD Flags
          28. WM_NCHITTEST Codes
          29. WH_SHELL Codes for the HSHELL_APPCOMMAND Hook Code
          30. WM_SIZE Message wParam Values
          31. WM_SYSCOMMAND Message wParam Constants
          32. XButton Constants
    9. Index
    10. About the Author
    11. Colophon
    12. Copyright

    Product information

    • Title: Subclassing and Hooking with Visual Basic
    • Author(s): Stephen Teilhet
    • Release date: June 2001
    • Publisher(s): O'Reilly Media, Inc.
    • ISBN: 9780596001186