VB Shell Programming

Creating a Context Menu Handler

Let’s put all of this into action and actually implement a context menu handler for the .rad file. We’ll add a context menu item that displays the noise an animal makes in a message box. The menu item itself will be displayed in the format (Animal Name) Noise . Animal Name will be determined from the .rad file in question. Let’s begin.

Type Library

The first step to creating the .rad file context menu handler is to compile the type library containing the interface definitions and constants that will be needed from VB. Constants and UDTs will also be put into the type library with their associated interfaces. But only the groups of constants that are needed will be put in the library. For instance, we need the menu constants MF_BYPOSITION, MF_STRING, and MF_SEPARATOR. Therefore, the library will contain all of the MF_ constants. We don’t need any of the menu state constants (MFS_ ), so they will not be included with the library.

The complete listing for the type library that will be used throughout the course of this book can be found in Appendix A. To compile the library, you need to have MKTYPLIB in your path. MKTYPLIB takes one argument on the command line, the name of the ODL file containing the type library definition. To compile, simply type:

mktyplib vbshell.odl

from the command line. If everything is in order, this should produce a file named vbshell.tlb. This is the type library.

To use this library from Visual Basic, you should select Project → References . . . from the main menu. You should then browse to the location of the .tlb file and select it. This will do two things. First, it will register the type library at that location; second, it will make it available to the References dialog for all future projects.

The Project

The context menu handler begins life as an ActiveX DLL project called RadEx. Our first step is to register the type library so that interface definitions are available for us to implement. That is done by selecting Project → References from VB and then Browse (the library is not registered, so it will not be in the list). Navigate to the library that is associated with this chapter and add the reference. The library will be available in the References list box from this point on.

Next, add the class that will implement the handler to the project. Call this class clsContextMenu. With the class added to the project, IShellExtInit and IContextMenu can be implemented as follows:

Option Explicit

Implements IContextMenu
Implements IShellExtInit

Implementing IShellExtInit

Let’s implement IShellExtInit::Initialize first. Notice that, in the code shell that Visual Basic creates for the Initialize method, the parameters are prefixed with the name of the library in which their definitions are located:

Private Sub IShellExtInit_Initialize( _
   ByVal pidlFolder As VBShellLib.LPCITEMIDLIST, _
   ByVal pDataObj As VBShellLib.IDataObject, _
   ByVal hKeyProgID As VBShellLib.HKEY)

In some cases, you might want to add a private variable to your class to hold the IDataObject reference passed in by the shell, since from it you can determine how many files are selected in the user interface and what the names of those files happen to be. We will use IDataObject to get the selected files from IShellExtInit::Initialize immediately, but it may be preferable to wait until a menu item is actually selected before the selected files are determined (possibly for performance reasons). In this particular case, saving the IDataObject reference is not necessary. Rather than hold a reference to IDataObject, we will use pDataObj directly and build an array containing the names of the selected files. This array will be kept as private data. The entire listing for the Initialize method is shown in Example 4.4.

Example 4-4. Implementing IShellExtInit::Initialize

'handler.bas

Public Declare Function DragQueryFile Lib "shell32.dll" _ 
    Alias "DragQueryFileA" (ByVal HDROP As Long, _ 
    ByVal pUINT As Long, ByVal lpStr As String, _ 
    ByVal ch As Long) As Long

Public Declare Function ReleaseStgMedium Lib "ole32.dll" _ 
    (pMedium As STGMEDIUM) As Long


'clsContextMenu.cls

Option Explicit

Implements IContextMenu
Implements IShellExtInit

Private m_sFiles(  ) As String
Public m_nFiles As Byte

Private Sub IShellExtInit_Initialize( _ 
    ByVal pidlFolder As VBShellLib.LPCITEMIDLIST, _ 
    ByVal pDataObj As VBShellLib.IDataObject, _ 
    ByVal hKeyProgID As VBShellLib.HKEY)

    Dim FmtEtc As FORMATETC
    Dim pMedium As STGMEDIUM
    Dim i As Long
    Dim lresult As Long
    Dim sTemp As String
    
    With FmtEtc
       .cfFormat = CF_HDROP
        .ptd = 0
        .dwAspect = DVASPECT_CONTENT
        .lindex = -1
        .TYMED = TYMED_HGLOBAL
    End With
    
    pDataObj.GetData FmtEtc, pMedium
        
    m_nFiles = DragQueryFile(pMedium.pData, &HFFFFFFFF, _
               vbNullString, 0)
    
    ReDim m_sFiles(m_nFiles - 1)
    
    For i = 0 To (m_nFiles - 1)
        sTemp = String(1024, 0)
        lresult = DragQueryFile(pMedium.pData, i, sTemp, _            
                                Len(sTemp))
        If (lresult > 0) Then
           m_sFiles(i) = Left$(sTemp, lresult)
        End If
    Next
    
    ReleaseStgMedium pMedium

End Sub

There’s quite a bit going here, so let’s just take it from the top, starting with the call to GetData. GetData takes two parameters: an [in] parameter containing a pointer to a FORMATETC structure, and an [in, out] parameter that returns a pointer to a STGMEDIUM structure. The function is called like so:

pDataObj.GetData FmtEtc, pMedium

The parameters are as follows:

FORMATETC

FORMATETC is a generalized clipboard format used by OLE wherever data format information is required. The structure contains the clipboard format, a pointer to a target device, the view of the data, how much of the data should be transferred, and the medium used to transfer the data. The members of the structure are assigned values in the following code fragment from Example 4.4:

With FmtEtc
        .cfFormat = CF_HDROP
        .ptd = 0
        .dwAspect = DVASPECT_CONTENT
        .lindex = -1
        .TYMED = TYMED_HGLOBAL
    End With

In this case, the data transferred will be a handle to a drop structure (our list of files) specified by CF_HDROP. The target device (specified by ptd ) is 0, because we don’t care about its value; it’s actually device-independent. DVASPECT_CONTENT means we want the actual data. A clipboard format can support more than one aspect or view. Here, we don’t need a view, we just need the data. lindex is unimportant to the discussion. Last is the TYMED_HGLOBAL flag, which means the transfer will take place using global memory (as opposed to a file or structured storage objects). The TYMED member specifies which member of the STGMEDIUM union will be valid.

Pointers

There are three undocumented functions that allow the use of pointers from Visual Basic. These functions are VarPtr, StrPtr, and ObjPtr. VarPtr will return the address of all VB datatypes (even UDTs), except for Strings. StrPtr is used to get the address of a String. ObjPtr is used to get the address of an Object. Internally, these three functions are all mapped to one function, VarPtr. But use these functions with caution. StrPtr and VarPtr will not return the same value for a String. StrPtr returns a pointer to the Unicode string value, and VarPtr returns the address where VB stores the pointer to the Unicode BSTR. For more information on VB’s undocumented pointer functions, see Appendix B.

STGMEDIUM

The second parameter to GetData is a pointer to a STGMEDIUM union. The union is based on the type of medium, which in this case is TYMED_HGLOBAL (specified by FORMATETC). Therefore, under normal circumstances, the union member hGlobal would contain the handle to the drop structure. However, since this structure has been redefined, the pData member will always point to the data. This handle can be passed directly to the Win32 DragQueryFile function, which then allows us to find out how many files have been selected:

nFiles = DragQueryFile(pMedium.pData, &HFFFFFFFF, vbNullString, 0)

Passing DragQueryFile the value &HFFFFFFFF tells it that we want the number of files selected. We can also pass it a number between and the total number of files selected to get the name of the file itself.

The value for nFiles allows us to redimension our file array. DragQueryFile can then be called in a loop with the index of the requested file supplied as the second argument to the function. The filename (which is written to the buffer that passed as the third argument to the function) is retrieved and stored in the file array. If multiple files of different types are selected and the file with primary focus is a .rad file, our handler will still be called. But we have to filter these extraneous types if necessary. To do this, we can have IContextMenu::InvokeCommand loop through this array and process the context menu command for every valid file that is selected.

Here’s one last detail: the STGMEDIUM structure has been allocated by the call to GetData. It is common to see this structure populated by a “provider” outside of the code in which it is being used, as is the case in Example 4.4. This means freeing the memory is our responsibility, and that is what the final call to ReleaseStgMedium (a routine found in Ole32.dll ) is doing.

Implementing IContextMenu

The IContextMenu interface is responsible for displaying the text of the menu item, for showing help text associated with the menu item, and for defining the action to be performed if the menu item is selected. In this section, we’ll examine the code for the methods responsible for those operations.

GetCommandString

The source code for the GetCommandString method is shown in Example 4.5. GetCommandString is called by the shell for the purpose of retrieving help text for a context menu item. This help text is then displayed in the status bar. This method is notable in that this is the first time we have to worry about implementing a method that will run under both Windows 98 and Windows NT. As you might guess, this has to do with how both platforms deal with strings. Windows 98 uses ANSI strings internally; Windows NT uses Unicode. VB uses Unicode strings internally, regardless of what platform is being used. Confusing, to say the least.

The menu item in question is determined by the idCmd parameter passed in by the shell. uType indicates the flags that inform us of the information being requested. We will return the same string regardless of these flags. The only distinction we are interested in is whether the values should be ANSI or Unicode. (There are separate ANSI and Unicode versions of each constant stored to uType.) A buffer for the help string is provided through the pszName parameter. cchMax is the size of this buffer.

The ANSI portion of the listing uses StrConv to convert the string from Unicode to ANSI. From this point forward, a common tactic is used. The string is copied into a byte array, and its starting address is copied to the memory location provided by the shell.

Example 4-5. GetCommandString Listing

Private Sub IContextMenu_GetCommandString( _
    ByVal idCmd As VBShellLib.UINT, _ 
    ByVal uType As VBShellLib.UINT, _ 
    ByVal pwReserved As VBShellLib.UINT, _ 
    ByVal pszName As VBShellLib.LPSTRVB, _ 
    ByVal cchMax As VBShellLib.UINT)

    Dim szName As String
    Dim bszName(  ) As Byte
    
    Dim sMenuHelp As String
    
    Select Case idCmd
        Case 0 'Noise
            szName = "Display Animal Noise"
        
        'Other menu items would be added like so:

        'Case 1 'Menu item 2
        '    szName = "Menu Item 2"
        
        'Case 2 'Menu item 3
        '    szName = "Menu Item 3"
    End Select
    
    szName = Left$(szName, cchMax) & vbNullChar
    
    Select Case uType
        Case GCS_VERBA, GCS_HELPTEXTA, GCS_VALIDATEA
            If (szName <> "") Then
                bszName = StrConv(szName, vbFromUnicode)
                CopyMemory ByVal pszName, _ 
                           bszName(0), _ 
                           UBound(bszName) + 1
            End If
        Case GCS_VERBW, GCS_HELPTEXTW, GCS_VALIDATEW
            If (szName <> "") Then
                bszName = szName
                CopyMemory ByVal pszName, _ 
                           bszName(0), _ 
                           UBound(bszName) + 1
            End If
    End Select
    
End Sub

InvokeCommand

InvokeCommand is called when the shell is ready to execute the context menu command. Its source code is shown in Example 4.6. The implementation of this method is fairly straightforward. Of interest is the pointer to the CMINVOKECOMMANDINFO structure that is passed in by the shell. CMINVOKECOMMANDINFO is one of those structures that mean something different depending on the context in which it is used. Check the Platform SDK for full details on this one.

This structure, while weighty as far as information goes, contains only one member that is of interest to us: lpVerb. The low-order word of lpVerb contains the menu identifier of the command being invoked.

By the time the shell calls InvokeCommand, we already have an array of the selected files stored as private data within our component. This allows us to grab every file in a loop, to find out the animal type of the file with a call to GetPrivateProfileString, and to display the appropriate information.

Example 4-6. InvokeCommand Listing

Private Sub IContextMenu_InvokeCommand(ByVal lpcmi As VBShellLib.LPCMINVOKECOMMANDINFO)

    Dim cmi As CMINVOKECOMMANDINFO
    CopyMemory cmi, ByVal lpcmi, Len(cmi)

    Dim i As Long
    Dim sNoise As String
    sNoise = Space(255)
    
    If LOWORD(cmi.lpVerb) = 0 Then
        For i = 0 To m_nFiles - 1
            
            GetPrivateProfileString "Animal", _
                                    "Noise", _
                                    "Unknown", _
                                    sNoise, _
                                    Len(sNoise), _
                                    m_sFiles(i)
            
            MsgBox Trim(sNoise), vbOKOnly, "Animal Noise"
            
        Next i
    End If
    
End Sub

The LOWORD function is defined in handler.bas. There is also a HIWORD function thrown in for good measure. The two functions look like this:

Public Function LOWORD(ByVal lVal As Long) As Integer
LOWORD = lVal And &HFFFF&
End Function

Public Function HIWORD(ByVal lVal As Long) As Integer
HIWORD = 0
If lVal Then
HIWORD = lVal \ &H10000 And &HFFFF&
End If
End Function

QueryContextMenu

QueryContextMenu is used to add menu items to a file object’s context menu. Implementing IContextMenu::QueryContextMenu is going to be a tricky process. The Platform SDK states that this method must return a positive integer representing the menu identifier of the last menu item added plus one. You might have noticed that these interface methods are implemented as subs, not functions. Even though we are dealing with a sub, VB still returns a value for each of these methods: a if everything is okay or an error code that is available through the Err object. We have no direct access to the value returned from these methods.

There is a solution to this dilemma. We will write a replacement function for QueryContextMenu and put it in a code module located in the project. Then we will find the vtable entry for QueryContextMenu in our object (see Chapter 1). We will use the AddressOf operator, in conjunction with CopyMemory, and swap the two addresses. Our new function, QueryContextMenuVB, will be called instead of the class implementation. Of course, QueryContextMenuVB will be a function, and we can return any value we want. When the object is released, the two addresses will be swapped back for posterity’s sake. Our troubles are solved.

The addresses of the two functions need to be swapped as quickly as possible. Therefore, the Initialize and Terminate events (which are shown in Example 4.7 and Example 4.9, respectively) of the context menu handler class are used for this purpose.

Example 4-7. Swapping vtable Entries

Private m_pOldQueryCtxMenu As Long

Private Sub Class_Initialize(  )

    Dim pVtable As IContextMenu
    Set pVtable = Me
    
    m_pOldQueryCtxMenu = SwapVtableEntry(ObjPtr(pVtable), _ 
        4, AddressOf QueryContextMenuVB)
    
End Sub

A variable of type IContextMenu is set to Me. This gives us a pointer to the IContextMenu portion of the vtable. This memory location is copied into pVtable, effectively giving us a pointer to the IContextMenu portion of our object’s vtable. Then, SwapVtableEntry (shown in Example 4.8) is called with the address of the first method of IContextMenu (this is the portion of the vtable where IContextMenu begins), the relative position in the vtable of the method we want to replace (in this case, 4—we’ll see why in a few moments), and the address of the new function. One thing of interest in SwapVtableEntry is the call to VirtualProtect. VB has marked the object memory as protected. This call changes the access permissions, allowing us to swap the addresses.

Example 4-8. SwapVtableEntry Listing

Public Function SwapVtableEntry(pObj As Long, _ 
    EntryNumber As Integer, _ 
    ByVal lpfn As Long) As Long

    Dim lOldAddr As Long
    Dim lpVtableHead As Long
    Dim lpfnAddr As Long
    Dim lOldProtect As Long

    CopyMemory lpVtableHead, ByVal pObj, 4
    lpfnAddr = lpVtableHead + (EntryNumber - 1) * 4
    CopyMemory lOldAddr, ByVal lpfnAddr, 4

    Call VirtualProtect(lpfnAddr, 4, _ 
                        PAGE_EXECUTE_READWRITE, _ 
                        lOldProtect)

    CopyMemory ByVal lpfnAddr, lpfn, 4
    Call VirtualProtect(lpfnAddr, 4, lOldProtect, lOldProtect)

    SwapVtableEntry = lOldAddr

End Function

How do we know where QueryContextMenu is located in relation to this address? Well, we can’t look at our class file for clues, because VB just displays all of the implemented methods in alphabetical order. This is not an accurate representation of our object.

To determine the vtable order of the method in question, look at the ODL listing. The methods are listed in the order in which they appear in the vtable. You can also use OLE View to get this information (should ODL be unavailable). Object Browser, however, does not provide it; it just lists the methods in alphabetical order. If you examine the IContextMenu interface definition in this manner, you will see that QueryContextMenu is the first method listed in the interface. Taking into consideration that the interface is derived from IUnknown, which contains three methods, QueryContextMenu is the fourth method. Thus, we pass 4 to SwapVtableEntry.

When the object terminates, the addresses can be switched back in the same manner, as shown in the class Terminate event handler in Example 4.9.

Example 4-9. Restoring vtable Entries

Private Sub Class_Terminate(  )
    Dim pVtable As IContextMenu
    Set pVtable = Me
    m_pOldQueryCtxMenu = SwapVtableEntry(ObjPtr(pVtable), _ 
        4, m_pOldQueryCtxMenu)
End Sub

QueryContextMenuVB

QueryContextMenuVB gives us some insight into just how a class works. We already know that a class keeps track of its member functions with the vtable. But once we are inside one of those member functions, how is it that we can have access back to the class? To the other methods? To Private and Public data members? Well, when a member function is called, a pointer to the class is also passed with the parameters to the function. VB (also C++) handles this behind the scenes, making everything look nice and smooth. C++ programmers refer to the parameter as the this pointer. VB can use this pointer to resolve all references back to the object.

QueryContextMenuVB must make allowances for this parameter, because it is not a part of a class; it is a function defined in a code module. This means we have to add our own this pointer to the parameter list. Example 4.10 shows how we can then define a local copy of clsContextMenu and use the this pointer to get a reference back to our class. This is really cool, because we don’t have to use a global variable to get at our class now.

Example 4-10. QueryContextMenuVB Implementation

'ContextMenu.bas

Public Function QueryContextMenuVB (ByVal this As IContextMenu, _ 
    ByVal hMenu As Long, _ 
    ByVal indexMenu As Long, _ 
    ByVal idCmdFirst As Long, _ 
    ByVal idCmdLast As Long, _ 
    ByVal uFlags As Long) As Long

    Dim ctxMenu As clsContextMenu
    Set ctxMenu = this

The main task of QueryContextMenuVB (which we seem to have ignored for a while) is to add menu items to the context menu. First, the circumstance in which the context menu is activated needs to be determined. This is accomplished with the uFlags parameter that is passed in by the shell. The following code fragment shows the various situations in which the context menu can be activated. The flag we are primarily interested in is CMF_EXPLORE:

If (uFlags And &HF) = CMF_NORMAL Then

    'Implement this for Drag-and-Drop handler.

ElseIf (uFlags And CMF_VERBSONLY) Then

    'This is a context menu for a shortcut item.

ElseIf (uFlags And CMF_EXPLORE) Then

    'Right-click on file in Explorer.
    'This is what we are interested in for our context
    'menu.

ElseIf (uFlags And CMF_DEFAULTONLY) Then

    'Indicates a default action is being performed (typically a
    'user is double-clicking on the file).

End If

Once it has been determined that files have been right-clicked in Explorer, the context menu item can be added accordingly. The menu item added is based on the number of files selected and the type of animal represented by the file. The animal type is determined with a call to GetPrivateProfileString:

ElseIf (uFlags And CMF_EXPLORE) Then
        
    'Right-click on file in Explorer
        
    If ctxMenu.FileCount > 1 Then
        sMenuItem = "Bunches o' Animal noises"
    Else
        GetPrivateProfileString "Animal", _
                                "Type", _
                                "Unknown", _
                                sAnimal, _
                                Len(sAnimal), _
                                ctxMenu.FileName
                                  
        sAnimal = Trim(sAnimal)
        sAnimal = Left$(sAnimal, Len(sAnimal) - 1)
          
        sMenuItem = sAnimal & "Noise"
        
    End If

    Call InsertMenu(hMenu, _ 
                    indexMenu, _ 
                    MF_STRING Or MF_BYPOSITION, _ 
                    idCmd, _ 
                    sMenuItem)

    idCmd = idCmd + 1
    indexMenu = indexMenu + 1
   
    'If you want to add another item just repeat the following code.
    '
    'sMenuItem = "Animal Name"
 	'Call InsertMenu(hMenu, _ 
    '                indexMenu, _ 
    '                MF_STRING Or MF_BYPOSITION, _ 
    '                idCmd, _ 
    '                sMenuItem)
    '
    'idCmd = idCmd + 1
    'indexMenu = indexMenu + 1
        
    'etc. , etc., etc.
    '
    'Do not increment idCmd for separators!
    'IndexMenu is always incremented.

    Set ctxMenu = Nothing

	'Lastly, the number of menu items added + 1 is returned.
    QueryContextMenuVB = indexMenu

Drag-and-Drop Handlers

Drag-and-drop handlers are displayed when a user drags a file with the right mouse button. They are really just specialized context menu handlers; therefore, they don’t get a chapter of their own.

To implement one, you would need to insert menu items based on the context menu flag being equal to CMF_NORMAL, and only equal to CMF_NORMAL. It cannot be AND ed with any other value. In addition, you would have to add the handler to the registry as shown in Figure 4.5.

The handler is also added to the approved shell extensions list in the registry (see Chapter 2).

Figure 4-5. Defining a drag-and-drop handler in the registry

Registration and Operation

Last but not least, the handler needs to be registered. As always, the file rad.reg that is included with this chapter’s downloadable code contains the appropriate registry entries. Example 4.11 contains the entire listing. Note that items in square brackets must exist on the same line (the listing is formatted to fit on the page).

Example 4-11. rad.reg

REGEDIT4

[HKEY_CLASSES_ROOT\.rad]
@ = "radfile"

[HKEY_CLASSES_ROOT\radfile]

[HKEY_CLASSES_ROOT\radfile]
@ = "Rudimentary Animal Data"

[HKEY_CLASSES_ROOT\radfile\shellex]

[HKEY_CLASSES_ROOT\radfile\shellex\ContextMenuHandlers]
@ = "RadFileMenu"

[HKEY_CLASSES_ROOT\radfile\shellex\ContextMenuHandlers\RadFileMenu]
@ = "{D4F9CECF-E84E-11D2-BB7C-444553540000}"

[HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Shell Extensions\Approved]
"{D4F9CECF-E84E-11D2-BB7C-444553540000}" = "RAD file context menu extension"

These are the same entries that were discussed in Chapter 3. Review the entries until you become familiar with them. It shouldn’t take too long. If you want to register the extension by hand, you will need to find the CLSID for the object. The easiest way to do that is to search under HKEY_CLASSES_ROOT for the programmatic identifier, or ProgID, of the extension object. The ProgID is formed by appending the class name to the project name with a period in the middle. So look for “RadEx.clsContextMenu,” and there should be a CLSID subkey with the needed value.

After you have registered the handler, kill off any instances of Explorer you might have running. The handler, which is shown in Figure 4.6, will be available with the next instance you run. There are sample .rad files included with the source of this book that you can use to test the handler.

Figure 4-6. Context menu in action