Subclassing and Hooking with Visual Basic

The AddressOf Operator

The AddressOf operator, first introduced in Version 5 of VB, gave developers limited access to pointers, a feature that VB effectively hides from them but that is essential to high-end development environments such as Visual C++. The AddressOf operator greatly increases the potential of a VB application. As we shall see, though, there are always bumps in the road when implementing more advanced functionality, and AddressOf has several of them.

AddressOf provides the VB developer with a simple way of using function pointers without relying on another language. A function pointer is simply a variable that contains the memory location of a single function. In other words, this variable points to a function. Now, instead of having to use the function name to call the function, we can instead use the function pointer to call the function.

A callback or callback function is the function which the function pointer references. Code that receives a function pointer can use it to call back (hence the name “callback”) to that function. Usually, these callback functions are small in size because they might be called many times per second and affect application performance.

Function pointers and callback functions are mainly used for asynchronous processing and with the enumeration application programming interface (API) functions. EnumWindows, EnumChildWindows, and EnumDesktopWindows are just some of Windows’ enumeration API functions. These functions each take a function pointer in their argument list. This function pointer is used to invoke a callback function for each item -- a window, in this case -- found by the API function. We’ll look at some examples of enumeration functions and asynchronous processing later in this section.

Using AddressOf

The rules defining how AddressOf must be used greatly limit its functionality and make it far less powerful than many VB developers had originally hoped. It seems that Microsoft’s plans for introducing VB developers to function pointers was primarily meant to allow access to Windows API functions that were previously unusable. Great, but what about using function pointers within a pure VB application? By this I mean calling a VB function and passing it a function pointer using AddressOf. This called function would accept the function pointer and use it to directly call a callback function. The answer is that AddressOf cannot be used in this manner. Disappointing, yes, but we still have the use of many API functions that were previously unusable.

There are several other limitations and problems to watch out for when using AddressOf in your applications. AddressOf must be placed immediately before a function name in an argument list for a called function. For example:

Call DLLFunction(hwnd, 0, AddressOf VBCallbackFunction)

This argument must be the name of a previously defined function. Because any kind of pointer is useless outside of the process that created it, it makes sense that the function that AddressOf precedes must be in the same process.

AddressOf can be used only with VB functions, subs, and properties. You cannot use it to get a pointer to an API function that you have declared in your code using the Declare statement. For example, you cannot do the following:

RetVal = EnumWindows(AddressOf EnumChildWindows, 0)

EnumChildWindows is a Windows-defined API function, not a VB function, sub, or property. This will cause an error when compiling your application.

It would be nice if the function whose pointer we pass using the AddressOf operator could reside anywhere in our VB application, but unfortunately, this is not permitted.

It is also important that any function that is passed a function pointer knows exactly how to call the callback function. The parameter lists of the callback function and the code that will be calling it must match exactly in number and in type.

Perhaps the most significant limitation of using callbacks is that the function pointed to by the AddressOf operator must reside in a code (BAS) module, rather than a form (FRM) or class (CLS) module. Of course, we can call functions within the CLS or FRM files from the callback in the BAS module, but our problem remains that we cannot cleanly package our callback function into an object.

Tip

Callback functions (including subclassed window procedures) must reside in a BAS module.

You might wonder why this limitation exists. FRM and CLS modules are considerably different from BAS modules: only one copy of the data in a BAS module is stored in the application’s process space, while FRM and CLS module data can be instantiated multiple times, with each copy of the object having its own data. Along with instantiating a FRM or CLS module, you also can destroy it by setting it to equal Nothing, as follows:

Set CObj = Nothing

This, in effect, removes all traces of that instance of the object from the process’s address space. Think how much trouble we could get ourselves into if we destroyed an object that contained the subclassed window procedure before we had replaced it with the original window procedure. The results would be disastrous. Similar problems would arise if we tried to use SetWindowLongPtr to insert the subclassed window procedure from an object that had not been created.

Within the BAS module, the callback procedures should be defined as Public. A public function in a BAS module is always visible from anywhere in the application. A public function in a FRM or CLS module is only visible when you have successfully created an object variable referencing that object (FRM or CLS module).

Tip

Define a callback function (including a subclassed window procedure) as Public.

There is one last problem with all owing callback functions to reside in FRM or CLS modules: they use vtables to get to their functions. A FRM or CLS module is basically a Component Object Model (COM) object -- that is, they adhere to the COM standards. This means that an extra level of abstraction exists to get to that object’s public functions. BAS modules, because they are not COM objects, do not have this extra layer of abstraction, and therefore their public functions are directly accessible from anywhere in the application’s code.

Finally, it’s important to understand that, when you pass a function pointer using AddressOf, you are arranging for some routine (usually a function in the Win32 API) that’s external to your application to temporarily pass flow control to a routine in your own application (that is, to the callback function). Because from your point of view this external routine is a black box that’s beyond your control, you should not raise any errors in the callback function that are propagated back to the calling routine. You should use On Error Resume Next to bypass the error. If necessary, you can check the Err object to see if an error has been raised while still inside the VB callback function. If one has been raised, you should handle it immediately, clear the error, and continue on.

Tip

Use On Error Resume Next for error handling in the callback function.

Callbacks and Enumeration Functions

To see how callback functions work with the Win32 enumeration functions, let’s use EnumChildWindows in a simple example. EnumChildWindows is declared in VB in the following manner:

Public Declare Function EnumChildWindows Lib "user32" Alias "EnumChildWindows" _
		(ByVal hWndParent As Long, ByVal lpEnumFunc As Long, _
		ByVal lParam As Long) As Long

Its parameters are:

hWndParent

The handle to the parent window whose child windows we want to enumerate

lpEnumFunc

A pointer to a callback function

lParam

Any other data that needs to be sent to the callback function

To use this function, a callback procedure needs to be written. The callback function for this example will be called EnumProc. The documentation for EnumChildWindows also describes this callback function in detail. It must have the prototype:

Public Function EnumProc(ByVal hWnd As Long, lParam As Long) As Long

This function takes two arguments. The first, hWnd, is a handle to a window in the enumeration list. The second argument is lParam, which receives a developer-defined value. Because it is passed by reference, this value also is passed back to the calling routine. Using this argument, one can pass information into and out of this callback function. To continue enumerating windows, this function should return TRUE; to stop enumerating them, it should return FALSE. Thus, a very simple EnumProc callback function appears as follows:

Public Function EnumProc(ByVal hWnd As Long, lParam As Long) As Long
	'Do work here

	EnumProc = True       
End Function

Now that we have a callback function, we can write the code to call the EnumChildWindows function. The first argument to this function is AddressOf EnumProc. This evaluates to a function pointer, which points to the EnumProc function. The second argument is zero, which is passed in to the lParam argument for the EnumProc function:

Sub Main(  )
	RetVal = EnumChildWindows(AddressOf EnumProc, 0)
End Sub

When this program is run it will call EnumChildWindows. This function, in turn, calls the EnumProc callback function once for every top-level window currently running in the system. When EnumWindows is finished processing, it returns control to the Main function.

Callbacks and Asynchronous Processing

Asynchronous processing is different from an enumeration function. Asynchronous processing allows code in the main application to call a function and then immediately return and continue execution of the main application’s code before the function’s code has finished processing. With it you can write code that allows a user to perform some time-consuming task, such as sorting a large amount of information, and the user will be able to continue using the same application without waiting for the sorting to finish. This gives the user the illusion of a fast application even though the sorting might take quite some time.

A callback function can be called during the sorting process to determine if the user has canceled the action. In processing involving large nested loops, you can use a function pointer to call a callback function, which determines if the user has clicked a Cancel button. If so, the callback function returns a status code informing the sorting routine that the user wants to cancel the sorting operation. The operation is canceled and the application proceeds onward.

A callback function also could be called to inform the application of the sorting operation’s progress. The application could then update status information displayed to the user that keeps the user informed of how much work still needs to be done by the sort routine.

Figure 4-1 illustrates the order in which functions are called for a VB application implementing asynchronous processing. In step 1, a function (Main) in the BAS file calls a function in a dynamic link library (DLL) and passes it a function pointer to the CB callback function, also within the BAS file. In step 2, the DLL function (DLLFunct) uses this function pointer to call the function CB in the BAS file during its processing. This function might notify the main application of the status of the DLLFunct function or determine if the user wants to cancel its operation. In step 3, the CB function finishes processing and immediately returns control to the DLLFunct function. In step 4, the DLLFunct function returns control to the Main function in the BAS file.

Figure 4-1. A diagram of how function pointers work in VB with the order of events

As I mentioned in Chapter 2, the lpfnWndProc member of the window class structure is a function pointer to the window procedure. The lpfn prefix tells us that this is a Long pointer to a function.

There is one problem with the application and DLL in Figure 4-1. The application stops executing as soon as it calls DLLFunct from the BAS file. The application is waiting for the DLLFunct function to return. The code in the callback function, CB, will still execute every time it is called, but for all practical purposes the VB application is waiting for the DLL to finish execution. To solve this problem and make the application truly asynchronous, we need to start a new thread in the DLL and use this thread to execute the DLLFunct function code. The only change to Figure 4-1 is the addition of a new function that is called from the Main subroutine in place of the DLLFunct DLL function. This new function would create a new thread, run the DLLFunct code on this thread, and immediately return control to the Main subroutine in the BAS file. With the addition of this new function, our VB application can continue running as if the DLLFunct procedure had never been called.

AddressOf and Subclassing

With subclassing, we will be using the AddressOf operator to get a function pointer to our new window procedure. This new window procedure is the callback function that the window message loop calls first, after receiving a message. The code required to implement AddressOf is quite simple. First you code the callback function:

Public Function NewWndProc(ByVal hWnd As Long, _
				 ByVal uMsg As Long, _
				 ByVal wParam As Long, _
				 ByVal lParam As Long) as Long
	'Your code goes here
End Function

This function will eventually serve as our new subclassed window procedure. Next we use the AddressOf operator to pass a pointer referring to this function into the Win32 SetWindowLongPtr function. The SetWindowLongPtr function call will look like this:

m_lOrigWndProc = SetWindowLongPtr(hWnd, GWLP_WNDPROC, AddressOf NewWndProc)

SetWindowLongPtr is described in detail in Chapter 3. This single line of code effectively subclasses the window that has hWnd as its handle. It accomplishes this by replacing the function pointer to the original window procedure with a function pointer to our NewWndProc function, acquired by using the AddressOf operator. The SetWindowLongPtr function will then return the pointer to the original window procedure to the calling function.

This window is now subclassed. Any messages sent to this window will be sent to the NewWndProc function. Instead of simply calling this function a callback function, it is usually called a subclassed window procedure. Any window procedure or subclassed window procedure is just a callback function used specifically to process window messages. This is why they are called window procedures instead of callback functions.

As we noted, any routine that is passed a function pointer must know exactly how to call the callback function. This is not too hard in the case of subclassing because all window procedures take the same number and type of arguments. If we deviate from this, our code will not work and will eventually crash the process that it is running in. It is not only necessary to match the function parameters, but it is also wise to understand how the calling function and the callback function work. This knowledge will help to prevent passing bad data to a callback function, which could cause problems ranging from simple logic errors to more troublesome General Protection Fault (GPF) problems. This also allows us to correctly handle data returned from a callback function.