Subclassing and Hooking with Visual Basic

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Windows is a message-based system. This means that every action you take while using the system creates one or more messages to carry out the action. These messages are passed between objects within the system. These messages also carry with them information that gives the recipient more detail on how to interpret and act upon the message.

Clicking a button control provides a good messaging example. This produces not only the message for the mouse button click, but also a wide array of other messages. These include messages to repaint the button in its depressed state, notification messages to inform other objects of the button's change in state, messages to determine the state of the mouse cursor, as well as others. Even a simple act such as moving the mouse or pressing a key on the keyboard can produce an astonishing number of messages.

In addition to communicating user actions, Windows also uses messages internally to do housekeeping. Messages need to be sent to update the time and date, to notify other objects of a change in state, and even to notify applications when system resources are exhausted.

The Windows messaging system is the heart of the operating system. As a result, the messaging system is very complex.

Subclassing and the Windows hooking mechanism operate on messages within the messaging system. This makes subclassing and hooking two very powerful techniques. With them, we can manipulate, modify, or even discard messages bound for other objects within the operating system and, in the process, change the way in which the system behaves. As you might already have guessed, a thorough understanding of the messaging system is critical to mastering the techniques of subclassing and hooking.

Along with this power comes responsibility. It is up to the developer to make sure that he or she is using these techniques correctly. Windows is very unforgiving if these techniques are used incorrectly.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Subclassing techniques deal with intercepting messages bound for one or more windows or controls. These messages are intercepted before they can reach their destination window. The intercepted message can be left in its original state or modified. Afterward, the message can be sent to its original destination or discarded.

By intercepting messages in this manner, we can have a powerful influence on how the window or control will react to the messages it receives. Consider, for example, right-clicking the Visual Basic (VB) text box control. This action causes a default pop-up menu to be displayed containing the following menu items: Undo, Cut, Copy, Paste, Delete, and Select All. Replacing this menu with one of our own is a fairly simple task using subclassing. Subclassing has many other uses as well, such as:

Determining when a window is being activated or deactivated and responding to this change
Responding to new menu items that are manually added to the system menu of a window
Displaying descriptions of menu items as the mouse moves across them
Disallowing a user to move or resize a window
Allowing a user to move or resize a window within specified boundaries
Determining where the mouse cursor is and responding accordingly
Modifying the look of a window or control
Changing the way a combo box operates
Determining when the display resolution has been changed
Monitoring the system for a low system-resource condition
Modifying or disallowing keystrokes sent to a window or control

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

The window hooking mechanism, or hooks, also deals with intercepting messages, but at a much broader scope than subclassing. Hooking allows us to intercept messages at various set points within the operating system. For example, we can intercept a message before and after a window has processed it.

There are several different kinds of hooks, each with their own special purpose and location within the operating system. They are:

WH_CALLWNDPROC
WH_CALLWNDPROCRET
WH_CBT
WH_DEBUG
WH_FOREGROUNDIDLE
WH_GETMESSAGE
WH_JOURNALPLAYBACK
WH_JOURNALRECORD
WH_KEYBOARD
WH_KEYBOARD_LL
WH_MOUSE
WH_MOUSE_LL
WH_MSGFILTER
WH_SYSMSGFILTER
WH_SHELL

Hooks, unlike subclassing, can have an application scope or a system-wide scope. By this, I mean a single hook can intercept specific messages within a single application, or it can be set up to intercept those same messages for all applications running in the system. Hooks give us control over the system, which cannot be achieved with subclassing. The following are just a few of the uses for hooks:

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Along with using these advanced techniques, effectively implementing subclassing and hooking in our development work requires that we employ debugging tools beyond the capabilities of the VB debugger.

While developing the projects for this book, I used several software utilities as well as other professional applications that I built. Although you can successfully build applications that subclass various windows or that hook into certain message streams without these utilities, I do not suggest doing so. These utilities give you, the developer, a valuable insight into what is happening inside the system while running your projects in the VB integrated development environment (IDE) and especially at runtime. You will be able to see things operate in a way that is unavailable to you by just using the Visual Basic or Visual C++ development environments.

I would go as far as saying that some of these utilities are necessary to understand how subclassing and hooks work. Otherwise, you will only be blindly plugging code into an application, not fully understanding why you are doing it and what is happening behind the scenes. When the application locks up, debugging it will be frustrating and possibly futile. What I am stressing here is that we, as programmers, must aspire to have an understanding of what we are doing. Without this understanding we cannot hope to reach the more advanced areas of our discipline. Having an understanding of how subclassing and hooks work and interact with the rest of the Windows system will allow you to build successful applications.

I will describe the utilities that I use in the following sections. Although this book will not include a tutorial for operating these utilities, there is some very good documentation in the Microsoft Developer Network Library (MSDN) for Spy++ and PView. The NuMega tools come with their own documentation. Note that some of these tools display different information depending on which operating system you are using (e.g., Windows 9x, NT, or 2000).

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

The techniques presented in this book make extensive use of the Win32 API and pointers. As you know, VB does not give us direct access to pointers. Instead we must use API functions to convert these pointers into information that VB can use.

As with using pointers in C, we must also take care when handling pointers in VB. Failing to do so will result in your application behaving unpredictably or crashing.

Incorrectly setting up and calling Win32 API functions is another source of problems. To function correctly, API functions must not only be declared correctly in VB, but also have their arguments passed in properly.

I will not be covering in any great detail how to set up and call Win32 API functions in this book. It is up to you to make sure the API functions that you use are declared and used properly. For more information on the topic of Win32 API functions, you can read Steven Roman's book entitled Win32 API Programming With Visual Basic, published by O'Reilly & Associates.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Before delving into the guts of subclassing and using hooks, we must first learn how the Windows messaging system works. This chapter is not going to teach you all the details of processes, threads, or some of the other Windows objects encountered while programming the operating system. Instead, I will concentrate on the particulars of the Windows operating system only insofar as they apply to the subject of this book.

Understanding how the Windows messaging subsystem works is a necessary first step toward learning how to correctly design and implement subclassing and hooks through Visual Basic (VB). In fact, this holds true for using subclassing or hooks in any language, but it is more important in VB because VB shields the developer from going too deeply into the Windows internals. This is a double-edged sword. On one hand, VB makes it very simple to construct applications that might take much more time and effort in a different language such as C++. On the other hand, not being able to easily get intimate with the lower levels of the Windows operating system makes the task of debugging and implementing advanced functionality into our applications more difficult. Knowledge of what is happening in the system, where it is happening, and why it is happening is essential to constructing and debugging your applications.

Tinkering with the Windows messaging system is neither straightforward nor easy. One wrong line of code, one misplaced pointer -- even exiting your application early -- could easily bring down the entire system. Creating invalid page faults as well as freezing an application or your entire system is easy to do when adding subclassing and/or hooks to an application. This is the main reason why some developers shy away from using these tools and sometimes even downplay their usefulness. Armed with the information in this book, you will be able to consistently use these tools to add advanced functionality to your applications with a minimum of pain and confusion.

With that said, this chapter will focus mainly on explaining the underlying Windows messaging system: what is it, how it works, as well as what makes up a message and how to interpret it. This will give you, the developer, a solid foundation on which to build throughout the rest of the chapters.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

This section focuses on the areas that are of greatest interest to us in using subclassing and hooks effectively. If you are interested in learning more about these subjects, pick up a copy of Win32 API Programming with Visual Basic by Steven Roman. He gives these topics a very thorough explanation.

All windows are related in some way to one or more other windows. The most common type of window relationship is the parent-child relationship. Other types of relationships include owner-owned and top-level windows. Although information on the relationship of one window to another is not a requirement to understanding subclassing or hooks, it is very helpful. This information will come into play more as I discuss subclassing the common dialog boxes and as I get into the specifics of hooks, such as the WH_SHELL hook. For example, the WH_SHELL hook only provides information on top-level, unowned windows.

Central to defining the relationship among windows is the concept of Z-order. When windows are drawn on the screen, only one window can be active at any time. This active window receives user input through the mouse and keyboard. This window also overlaps all other displayed windows on the screen. The windows below the active window are stacked one on top of the other; this is illustrated in Figure 2-1. You can think of this stack of windows as being similar to a stack of paper. The piece of paper on the top of the stack is the topmost piece of paper. The next piece of paper is located below the topmost piece, the third piece down is located below the second one, and so on and so forth.

Figure 2-1: The window Z-order

A Z-order defines where a window is currently at in the stack of windows. Making a window the active window will place it at the top of the Z-order. Using the Z-order, we can determine which window will be given the focus when the current window is minimized. The window given the focus will be the next window down, starting from the top of the Z-order.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

The messaging system is at the core of the Windows operating system. Without messages, the operating system would be about as useful as a pile of rocks. Subclassing and hooks operate on the messaging system; this is why it is so important to understand messaging as it applies to the Windows operating system. In this section, I will discuss each separate mechanism within the messaging system to give you an idea of how these pieces relate and operate together to send messages from a source to a destination window. Figure 2-4 shows how the messaging system works.

Figure 2-4: The Windows messaging system

The operating system upon bootup creates one thread for itself, called the raw input thread (RIT). There is only one of this type of thread in the system. This thread contains a system message queue used to receive hardware events such as mouse clicks and keyboard keypresses. It can collect events from these hardware devices through device drivers. A device driver is basically a type of DLL that acts as the translator between the hardware device and the Windows system. This allows hardware devices to communicate with the system. A device driver sits and waits for input from the hardware device for which it is associated. After it gets some type of input, it interrupts the system and sends a hardware event to the system message queue on the RIT. The RIT takes the hardware event off of its system message queue and converts it to a standard Windows message. The RIT posts the message to the correct thread's (or threads') message queue. The last step in the process is for the thread's message queue to deliver the message to the correct window procedure.

Let's take, for example, a mouse-button click that is generated by the user. The mouse driver gathers information about the mouse and places it into a MSG structure, defined as follows:

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

When implementing subclasses and hooks, performance is a critical factor. Too much code executing within a hook or a subclassed window could easily bring performance down below an unacceptable level. Keep this in the front of your mind when writing the code to handle subclassing and hooks. System-wide hooks have the most potential to degrade performance. I will discuss where these performance bottlenecks can arise as we progress through this book.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

This chapter will introduce how subclassing and hooks operate. I will use bits and pieces of code, instead of complete examples, so that we can focus on the basics of hooking and subclassing. The finer details of the code will be covered in the remainder of this book.

This chapter is broken up into two sections. The first section deals with subclassing and how it works, as well as what it is useful for. The second section deals with the hooking mechanism in a similar fashion. Of the two languages I use, I will focus as much as possible on Visual Basic (VB) and use Visual C++ only when VB has reached its limits.

The idea behind subclassing is simple; implementing it is not so simple. Subclassing is, in simple terms, the creation of a new window procedure, which is inserted into the message stream just before the default window procedure that every window initially starts out with. I use the term message stream to denote the path a message takes from its source to its destination.

This is not to say that subclassing is not dangerous when implemented incorrectly or without regard to other processes running in the operating system. But by understanding the messaging system and following its rules, we can safely use subclassing.

The dangers of subclassing are what stop many programmers from learning and using subclassing in their projects. The foremost danger with subclassing is causing a General Protection Fault (GPF). GPFs are critical errors that will either cause your application/system to stop functioning or cause the application/system to shut down. If you are running an application in the VB IDE and it causes a GPF, not only is the application shut down, but also the IDE. This is because both are running in the same process, and a GPF will cause that process to terminate.

Several things cause GPFs. They include:

Using a pointer variable that points to an invalid location in memory

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

The idea behind subclassing is simple; implementing it is not so simple. Subclassing is, in simple terms, the creation of a new window procedure, which is inserted into the message stream just before the default window procedure that every window initially starts out with. I use the term message stream to denote the path a message takes from its source to its destination.

This is not to say that subclassing is not dangerous when implemented incorrectly or without regard to other processes running in the operating system. But by understanding the messaging system and following its rules, we can safely use subclassing.

The dangers of subclassing are what stop many programmers from learning and using subclassing in their projects. The foremost danger with subclassing is causing a General Protection Fault (GPF). GPFs are critical errors that will either cause your application/system to stop functioning or cause the application/system to shut down. If you are running an application in the VB IDE and it causes a GPF, not only is the application shut down, but also the IDE. This is because both are running in the same process, and a GPF will cause that process to terminate.

Several things cause GPFs. They include:

Using a pointer variable that points to an invalid location in memory
Trying to read from memory that is not accessible from the application (e.g., in a separate process memory space)
Trying to write to memory that is not accessible from the application (e.g., in a separate process memory space)
Reading or writing past an array or string boundary
Incorrectly calling a dynamic link library (DLL) function

GPFs are elusive problems for the VB programmer to track down; in fact, they're even elusive problems for the C++ programmer to track down. That is why using tools such as NuMega Bounds Checker and Microsoft Spy++ (which are discussed in Chapter 1) are necessary when tracking down hard-to-find problems such as these.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

The Microsoft documentation defines hooks in this manner:

A hook is a point in the system message-handling mechanism where an application can install a subroutine to monitor the message traffic in the system and process certain types of messages before they reach the target window procedure.

One more thing should be added to this definition. The addition I have made is in italics:

A hook is a point in the system message-handling mechanism where an application can install a subroutine to monitor the message traffic in the system and process certain types of messages before they reach the target window procedure, as well as after they are processed by the target window procedure.

The subroutine that is installed at a hook point is typically called a filter function. A filter function is analogous to a window procedure in that it can receive and process messages. When these filter functions are placed into the message stream at a hook point by using the SetWindowHookEx API function (which will be defined later in this chapter), it is called installing a hook. A hook point is a system-defined point in the message stream at which a filter function can be installed. These hook points cannot be changed.

Hooks are similar to subclassing in that both intercept messages, although this is where most of the similarities end. You might think of hooks as extensions to subclassing. If a problem cannot be solved with subclassing, look into using hooks to augment or even replace subclassing. It is highly possible that several hooks could be used in combination to solve a problem.

The fundamental characteristic of all types of subclassing is that a new developer-defined window procedure is created which intercepts messages before the original window procedure has a chance to receive them. This new window procedure then decides how to handle the message and whether it will be passed on to the original window procedure.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

This chapter takes you deeper into the subject of subclassing, starting off with a discussion of the AddressOf operator. A fair amount of discussion is given to this operator because it plays a pivotal role in using Visual Basic (VB) to subclass windows. Presented next will be the two types of subclassing -- instance and global subclassing. Their similarities, differences, and applications are discussed at length. Code examples will be presented for both types of subclassing. These examples are meant to be for illustration only; such things as error handling code will be omitted for clarity.

As we progress through this chapter I will be placing the key points and rules of subclassing in bold type. These points and rules will be summarized at the conclusion of the chapter.

The AddressOf operator, first introduced in Version 5 of VB, gave developers limited access to pointers, a feature that VB effectively hides from them but that is essential to high-end development environments such as Visual C++. The AddressOf operator greatly increases the potential of a VB application. As we shall see, though, there are always bumps in the road when implementing more advanced functionality, and AddressOf has several of them.

AddressOf provides the VB developer with a simple way of using function pointers without relying on another language. A function pointer is simply a variable that contains the memory location of a single function. In other words, this variable points to a function. Now, instead of having to use the function name to call the function, we can instead use the function pointer to call the function.

A callback or callback function is the function which the function pointer references. Code that receives a function pointer can use it to call back (hence the name "callback") to that function. Usually, these callback functions are small in size because they might be called many times per second and affect application performance.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

The AddressOf operator, first introduced in Version 5 of VB, gave developers limited access to pointers, a feature that VB effectively hides from them but that is essential to high-end development environments such as Visual C++. The AddressOf operator greatly increases the potential of a VB application. As we shall see, though, there are always bumps in the road when implementing more advanced functionality, and AddressOf has several of them.

AddressOf provides the VB developer with a simple way of using function pointers without relying on another language. A function pointer is simply a variable that contains the memory location of a single function. In other words, this variable points to a function. Now, instead of having to use the function name to call the function, we can instead use the function pointer to call the function.

A callback or callback function is the function which the function pointer references. Code that receives a function pointer can use it to call back (hence the name "callback") to that function. Usually, these callback functions are small in size because they might be called many times per second and affect application performance.

Function pointers and callback functions are mainly used for asynchronous processing and with the enumeration application programming interface (API) functions. EnumWindows, EnumChildWindows, and EnumDesktopWindows are just some of Windows' enumeration API functions. These functions each take a function pointer in their argument list. This function pointer is used to invoke a callback function for each item -- a window, in this case -- found by the API function. We'll look at some examples of enumeration functions and asynchronous processing later in this section.

The rules defining how AddressOf must be used greatly limit its functionality and make it far less powerful than many VB developers had originally hoped. It seems that Microsoft's plans for introducing VB developers to function pointers was primarily meant to allow access to Windows API functions that were previously unusable. Great, but what about using function pointers within a pure VB application? By this I mean calling a VB function and passing it a function pointer using

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

As we've seen, subclassing involves creating a particular kind of callback function, one that is called by a window's event loop. Hence, the general tips that we've presented for callback functions (such as defining the callback as a Public function in a BAS module and using On Error Resume Next for error handling in the callback function) apply as well to subclassed window procedures. In addition, though, two tips and pitfalls that are unique to subclassing are worth mentioning.

First, remember that all messages for a window are passed through its subclassed window procedure. As a result, the subclassed window procedure (as well as a hook's filter function) can potentially be called hundreds or even thousands of times per second. If large amounts of code are executed in the window procedure for many of the messages, the performance of the application will degrade considerably. This makes it critically important to avoid doing too much work within a callback function. Performing file I/O in either type of procedure is one type of operation that could take up an unusually long amount of time to finish because of relatively slow disk access. If such long processes are included within these callback functions, the results could be less than satisfactory.

Do as little work as possible in the subclassed window procedure.

Second, never use the DoEvents function inside the window procedure. DoEvents will halt processing to allow other queued messages to be processed. Using this function inside a window procedure stops processing for that particular message. The problem occurs when a new message is sent to the window procedure before the previous message can finish processing. The first message yields to the second message, the second message yields to the third message, and so on. The outcome of this is unpredictable because messages might be processed out of order, or not be processed at all.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

As we saw in Chapter 3, instance subclassing involves using the SetWindowLongPtr function to replace the window procedure of a specific window instance.

In this section, we'll write our first simple subclassing application. The application will have two buttons, one to subclass the VB form and another to remove the subclass. The only other control will be a multiline text box that will display messages as they are sent to this form. I will add some more functionality to this application in the example following this one.

Let's start with the layout of the form. It's rather simple; the nondefault properties are listed in Table 4-1, and the form is displayed in Figure 4-2.

Table 4-1: Nondefault Properties of Form and Controls for Subclassing Example
Object	Property Name	Property Value
frmCh4	Caption	"Chapter 4 -- Subclassing Example"
frmCh4	ClientHeight	3612
frmCh4	ClientLeft	48
frmCh4	ClientTop	336
frmCh4	ClientWidth

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Global subclassing revolves around the same principles as instance subclassing. With global subclassing, though, we are going to go one level deeper before performing the subclassing. By this I mean that the modifications to the window procedure function pointer will occur in the class itself, not the individual window created from the class. Only those windows created from the class after the subclassing has occurred will use the new subclassed window procedure. When I mention subclassing in this section, I am referring to global subclassing.

Instance subclassing affects a specific window and its window procedure. Global subclassing affects every window created from a window class that has had its class window procedure modified.

Unlike instance subclassing, global subclassing can capture the window creation messages -- and, more specifically, WM_CREATE and WM_NCCREATE. When a window is created, the WM_NCCREATE message is sent first to the window to finish creating its nonclient area in memory. The WM_CREATE message is sent next to finish creating the window's client area in memory. Note that the window is still not displayed on the screen at this point. Other messages still need to be sent to the window to position, size, and paint it on the desktop.

The reason that instance subclassing cannot capture these messages is that when instance subclassing occurs, the window has already been created. Therefore, the window creation messages have already been processed. Global subclassing, on the other hand, occurs before a window is created. The subclassed window procedure is in place at the point when the window is created. Therefore, all window creation messages are captured.

Global subclassing allows the window creation messages to be processed. Instance subclassing does not have this ability.

To see how global subclassing differs from instance subclassing, we'll examine the modifications needed to change the previous instance subclassing example (rolling up the form using the window procedure shown in Example 4-10) to perform global subclassing.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

This chapter presented a great deal of information about subclassing. Before proceeding on to the following chapters, make sure that you have a solid grasp of the concepts presented in this chapter. These concepts will be used throughout the rest of the book. To help with this, the following is a list of key points mentioned in the course of this chapter:

Use On Error Resume Next for error handling in the subclassed window procedure.
Do as little work as possible in the subclassed window procedure.
Stepping through a subclassing application in the IDE can be problematic.
Subclassed window procedures and other callback functions must reside in a BAS module.
Define the subclassed window procedure and any other callback functions as Public.
Do not end an application before removing all subclassed window procedures.
Do not use the End statement in your code or the Stop button in the VB IDE.
Never lose the function pointer to the original window procedure for the subclassed window.
Do not use the DoEvents function within any window procedure.
Thoroughly research the messages that you will be trapping in the subclassed window procedure. Each message has its own little quirks.
To allow the window to perform the default processing for a message, it must be passed on to the original window procedure (CallWindowProc) or to the default window procedure (DefWindowProc).
Instance subclassing affects a specific window and its window procedure. Global subclassing affects every window created from a window class that has had its class window procedure modified.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

This chapter will focus on subclassing, but in a more specific area. In this chapter I will show you how to subclass the Windows common dialogs. Subclassing is the term commonly used to describe what we will be doing in this chapter, but as we shall see, the term might not be entirely accurate. The Open, Save As, Print, Page Setup, Print Property Sheet, Font, Help, Find, Replace, and Color dialogs are all part of the standard dialogs that ship with the Windows operating system. These dialogs are found in comdlg32.dll. Most of them also are encapsulated in an ActiveX control called the Common Dialog control.

Visual Basic (VB) can use either comdlg32.dll or the Common Dialog control to create and manipulate these dialog boxes. Using the Common Dialog control hides much of the complexity of using the common dialogs. However, if you want to subclass a common dialog, you have to use comdlg32.dll directly. Throughout this chapter I will be using comdlg32.dll to create and manipulate the common dialog boxes that we will subclass.

You might wonder why you would want to subclass one of the Windows common dialogs, especially because modifying a common dialog is usually the exception rather than the rule. But sometimes modifying a common dialog box is a more efficient approach to solving a problem, especially when the developer needs more control over a user's interaction with a common dialog box. Instead of creating a whole new Save As or Print Setup dialog box from scratch, we can use the functionality of the Save As common dialog box and augment, customize, or modify its behavior. This saves the developer many hours of re-creating a common dialog by leveraging the prebuilt code within comdlg32.dll.

In this chapter, we'll build an example application (see Figure 5-1) that focuses on teaching the fundamentals of subclassing a common dialog box. The common dialog used in this application is the Save As dialog, which we'll modify to perform a simple function by adding four controls to it: a checkbox, a drop-down list box, a text box, and a button. (The button is the only control that serves no real purpose, except to display a message box. I just wanted to add it in this example to show you how to use a button control in a subclassed common dialog box.) Essentially, our sample application will transform the Save As dialog into a rudimentary Export As dialog box, as shown in Figure 5-2. This subclassed common dialog box will take a text file containing several columns of words or numbers and delimit each word or number with a character or characters of the user's choice. The drop-down list box contains a list of standard delimiters that the user can choose from.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Subclassing a common dialog box is not as easy as subclassing a window. A few extra steps are involved. The four main steps are:

Create a child dialog template resource.
Package this resource in a dynamic link library (DLL), optional
Create a hook procedure that will intercept messages for the common dialog box.
Use the API function in the comdlg32.dll file to create the modified common dialog.

I will be using the term dialog procedure instead of "window procedure" to describe the message processing function for the common dialog box. These two types of procedures, for all practical purposes, are the same. One small difference is that the dialog procedure sends all unprocessed messages to the DefDlgProc function, whereas the window procedure sends all unprocessed messages to the DefWindowProc function.

Before going any further with subclassing common dialogs, we will see how to use the comdlg32.dll APIs. The common dialog that we use for subclassing must be created with the comdlg32.dll API instead of with the ActiveX Common Dialog control (comdlg32.ocx). The consequence of using the DLL instead of the ActiveX control (OCX) is mainly added complexity: all the details that the OCX hid from us are now up to us to handle. The good thing is that there is no longer a performance penalty for having to load the comdlg32.ocx, and this file will not have to be shipped along with your final application.

The common dialog box relies on comdlg32.dll for much more that just its creation. This DLL provides the dialog procedure for the dialog (a dialog procedure does for a dialog what a window procedure does for a window). This DLL also contains the default dialog resources needed to describe the common dialog box. This is why we must add our own dialog template resource to the common dialog box instead of just modifying the common dialog box resource directly. The DLL also handles message routing within the common dialog boxes.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

After the resource file (.RES) is created, there are two ways to use it within a subclassing application in VB. The first way is to simply add the resource file to the application. The second way is to add the resource file to a DLL and load the resource from this DLL. There is a reason for using each method to incorporate a resource into a project. If an application is running in the IDE at design time, the resource must be in a DLL. When the application is running as a standalone EXE, the resource can be embedded into the EXE as a .RES file, or it can be contained within a DLL. This is a limitation of VB, not of the resource file.

No other application can access the dialog template resource if it is embedded within this application as a .RES file because they are running in separate processes. Placing a resource in a separate DLL allows multiple applications to access the same resources. A single application also can access multiple resource DLLs.

VB has three functions for loading and using resources: LoadResString, LoadResData, and LoadResPicture. Although LoadResString and LoadResPicture are functions that load particular kinds of data (strings and images, respectively) from resource files, LoadResData is a general-purpose function for retrieving items from resource files. Its syntax is:

LoadResData(resID, resType)

where resID is the identifier of the resource, and resType is an integer that identifies the type of resource that's being loaded. To load a dialog resource, you supply the LoadResData function with a resType value of 5, indicating that this is a dialog resource. Unfortunately, this method only returns an array of bytes, which you are then required to process through your own code. After processing this information, you have to create the dialog manually through your code.

Even though the LoadResData

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

The application we will create in this section subclasses the Save As common dialog box. This application has one form module called frmMain.frm, which is displayed in Figure 5-7. The nondefault properties of this form are listed in Table 5-5. The form contains a command button control called Command1 and a text box control called Edit1.

Figure 5-7: The main form of the sample application

Table 5-5: Nondefault Properties of frmMain and Its Controls
Control Name	Property Name	Property Value
frmMain	Caption	Ch5 - Common Dialog Subclassing Example
frmMain	BorderStyle	3-Fixed Dialog
frmMain	Top	0
frmMain	Left	0
frmMain	Height	3420
frmMain	Width	5760

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

The previous material covered subclassing the Explorer-style Open and Save As common dialog boxes. This section covers subclassing all other types of common dialog boxes. The techniques in this section pertain to these common dialogs:

Color
Font
Print
Print Property Sheet (Win2000)
Page Setup
Find
Replace

The foremost difference between the Open and Save As common dialogs and the remaining common dialogs is how the dialog template resource is created. With the Open and Save As common dialogs, a separate child dialog had to be created and integrated into the parent common dialog box. This is necessary because the dialog resource for these two common dialogs is embedded in the comdlg32.dll. Not so with these other common dialog boxes. Their dialog resources are available in C++ header files (*.h) and in resource (*.dlg) files. The header files contain all the control IDs for each control on these dialogs. The resource files contain the actual dialog resource. Table 5-7 lists all resource IDs, header files, and resource files that each common dialog box uses. These files are shipped with the Visual C++ development environment.

Table 5-7: Resource IDs, Header Files, and Resource Files of the Common Dialog Boxes
Dialog type	Resource ID	Resource File	Header File
Color

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

The Find and Replace common dialog boxes operate differently than all other common dialog boxes in one respect. These common dialogs are modeless. This means that code will continue executing in the calling procedure; it will not wait for the dialog to be closed. For a modeless dialog to work properly, Windows requires that the IsDialogMessage API function be used in the main message loop of the application. The IsDialogMessage API function basically does the work of the TranslateMessage and the DispatchMessage API functions, but for modeless dialog boxes. Therefore, if IsDialogMessage returns a TRUE value, it has processed the message and the TranslateMessage and the DispatchMessage API functions should not be called. The IsDialogMessage API function's main purpose is to provide default dialog keyboard processing for the modeless dialog box. Modal dialog boxes automatically support default dialog keyboard processing. Here is the list of default dialog keyboard keys processed by the IsDialogMessage API function:

ALT+mnemonic: Moves the focus to the first control in the tab order after the static control containing this mnemonic
DOWN: Moves the focus to the next control
UP: Moves the focus to the previous control
ENTER: Simulates clicking the OK button
ESC: Simulates clicking the Cancel button
LEFT: Moves the focus to the previous control

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|

Table of Contents