Practical JIRA Plugins

Chapter 4. Custom Field Searchers

Overview

The previous chapters discussed how custom fields work in JIRA. This chapter covers how searching for values in custom fields works, along with an example of a custom field searcher for the Currency custom field type that was created in A New Custom Field Type. All the source code for the examples in this chapter is available from https://marketplace.atlassian.com/41293.

When you configure a new custom field in JIRA you specify a name for the field and choose a custom field type. At the same time you can also optionally select a search template so that you can search for issues with a given value in the field.

For example, when you add a new custom field of the standard type “Multi Select”, the default search template is named “Multi Select Searcher”. This searcher is what controls what you see when you’re searching for issues in the Custom Fields section of the top-level Issue Navigator page. There may be more than one searcher available for each custom field type. If you select “None” for the searcher, then you won’t see any reference to this field in the Issue Navigator search fields.

When creating a new custom field type such as Currency, you may well be able to reuse an existing searcher. For example, a custom field type that just formats some text in a different way can probably reuse the “Free Text Searcher” class that’s already defined for the standard JIRA Free Text custom field type. Alternatively, you may want to create a new searcher and have your new custom field type use that, either using existing Velocity template files or some new templates created just for that searcher. The standard JIRA searchers are defined in system-customfieldtypes-plugin.xml in customfield-searcher elements.

You can also add a new searcher to an existing custom field type such as “Free Text” or “Select List”. This is useful if you want to change how searches on an existing custom field happen. You can’t directly change the searchers used for system fields but see Further Reading for a way to work around this.

Figure 4-1 summarizes the relationship between some of the custom field types and their searchers. The dotted lines indicate searchers that could be used by the custom fields.

Figure 4-1. Custom Field Types and Searchers

The next section covers some of the underlying mechanisms of how searchers work. An understanding of this is helpful when you come to implement and debug your own custom field searcher, but if you just want to quickly create a searcher for your own custom field type, skip ahead to the examples that start at the section A Simple Searcher.

How Searchers Work

For performance reasons JIRA almost never queries its underlying database directly. Instead, it creates indexes of the necessary data using the popular Lucene search tool with one Lucene Document object per JIRA issue. These indexes are updated when an issue changes or when the entire Lucene index is rebuilt. JIRA then uses these much faster Lucene indexes to search for the Documents that match a query and then returns the related JIRA issues.

A custom field searcher defines both what should be added to each Lucene Document in the index, and also the Lucene query to run against the index when searching for issues. Searchers also define what appears when results are grouped and the autocompletion prompts used when entering search values.

JIRA has its own query language named JQL (JIRA Query Language) which is used to generate these Lucene queries. The JQL syntax resembles the more familiar database query language SQL. You enter constraints in the Issue Navigator simple search screen, for example by selecting the TEST project and a status of Open. From these constraints JIRA creates a JQL query. The advanced search screen shows the JQL query directly, in this case:

      project = TEST and status = Open

It’s worth noting that you can also create plugins that provide new JQL functions for searching both fields and also the rest of JIRA’s data. However to change what is stored for each issue you need to use a searcher. See the section Further Reading for links to more information about this.

Methods for a Custom Field Searcher

JIRA custom field searcher classes extend the base class AbstractInitializationCustomFieldSearcher, which has a CustomFieldSearcherModuleDescriptor variable that contains the details of how the searcher was defined in atlassian-plugin.xml. This descriptor is also used to create the Velocity context, which is used to render how the searcher appears on a web page.

All custom field searchers also implement the CustomFieldSearcher interface. This interface and its parent interface IssueSearcher have the following methods.

void init(CustomField field)

This is where a searcher gets connected to a specific custom field. The init method must set four variables that are used by the other methods. These four variables are:

searcherInformation
searchInputTransformer
searchRenderer
customFieldSearcherClauseHandler

getDescriptor()

This method returns the CustomFieldSearcherModuleDescriptor variable, which is the class that handles creating HTML related to the custom field and the searcher.

SearcherInformation getSearchInformation()

This method returns a SearcherInformation object which is used to refer to information about this searcher. That object in turn has a method getRelatedIndexers that returns the FieldIndexer classes that add this field’s data to the Lucene Document in an index, as described in Adding Data to the Lucene Index.

SearchInputTransformer getSearchInputTransformer()

This method returns the SearchInputTransformer object, which is used to convert the value submitted from the Issue Navigator field to the format used in a SearchRequest for a query. Some of the class names of these transformers are unwieldy—e.g., AbstractSingleValueCustomFieldSearchInputTransformer.java.

SearchRenderer getSearchRenderer()

This method returns the SearchRenderer object which renders the HTML for the searcher by using the descriptor object. The search renderer uses a CustomFieldValueProvider object, which is what calls the CustomFieldType methods to get the $value object used in a Velocity template.

CustomFieldSearcherClauseHandler getCustomFieldSearcherClauseHandler()

Provides the object that handles the JQL clauses that this searcher creates. These Clause objects are eventually converted to a Lucene query. This object is also where the supported operators and data type of the searcher are stored. The ClauseValuesGenerator parameter in the constructor for this object is where the 15 autocompletion values for JQL are created for each field.

Another interface that is implemented by many searchers is SortableCustomFieldSearcher which makes sorting results much faster than just using a custom field’s SortableCustomField interface. This interface has just one method, which is getSorter.

The interface CustomFieldStattable is used if you want to be able to use the results from this searcher in gadgets and reports that summarize by a particular custom field. That interface is described in more detail in Statistical Searchers.

Adding Data to the Lucene Index

All kinds of data can be stored in a Lucene index, usually as a string. The value stored corresponds to the value of a custom field, but it could also be a value of a calculated field (Read-only and Calculated Fields) or something else entirely, as shown in the example in More Complex Searchers.

Caution

The value stored in an index only gets updated when an issue is edited. This is usually just fine, except if a calculated field is depending upon values that are not part of that same issue. For example, if a calculated field displays the sum of numbers in fields from three other issues, and one of the values changes, then the stored index value is now out of date and incorrect search results will occur. One approach to solving this is to have a service that updates the index values periodically. Another approach is to use a listener that reindexes specific issues when the values the field depends on changes.

The FieldIndexer object that is passed to the SearcherInformation constructor is what controls the values that are added to the Lucene index. The FieldIndexer object is only used when an issue has been changed, or when the whole Lucene index is being recreated by a JIRA Administrator or a scheduled JIRA job or service.

All FieldIndexer classes should extend AbstractCustomFieldIndexer which has three methods of interest. The first two methods are abstract and so must be implemented. In most FieldIndexer implementations they both call the third method to do the actual indexing work.

addDocumentFieldsSearchable(): This method is called when updating the index for a custom field that is visible in the given issue. The field’s value is usually added verbatim to the Lucene index but a modified version may be used instead.
addDocumentFieldsNotSearchable(): This method is called when updating the index for a custom field that is not visible in the given issue, for example if the custom field is not valid in the issue’s project. The value is still added to the Lucene index but is marked as not present in this issue.
addDocumentFields(): This is the main method called by the other two methods to get the field’s value from the issue and convert it to a String for storing in the Lucene index. The indexType parameter is important since it controls what Lucene does to the value before it is added to the index. This is also how more than one value for a field can be stored in given Lucene Document as described in More Complex Searchers.

Tip

If you want to see exactly what is in each Document in a Lucene index, download the Luke diagnostic tool from http://code.google.com/p/luke and run it with:

java -jar lukeall-1.0.1.jar

Choose any file in caches/indexes/issues under your jira.home directory, and check the “Open in Read-Only mode” checkbox.

You’ll see the Lucene Fields, which are the various pieces of information that can be searched on for an issue. The Documents tab has arrows that let you see each Lucene Document with the issue_id and key fields for its JIRA issue. The Doc. Id field is the unique id for each Lucene Document.

The Search tab lets you choose a default field on the right, add a value on the left, and then click Search to see the information about a specific JIRA issue. Or you can just enter key:TEST-123 on the left and click the Search button. Double-clicking on the result brings up the information about that issue’s values in the index.

Executing a Search

When you click the Search button to run a query, the JQL query is parsed to generate an object that implements the com.atlassian.query.Query interface. This Query object has two main methods: getWhereClause and getOrderByClause. The names of these methods reflect what a typical SQL select statement looks like, for example:

        select * from jiraissue where project=10100 order by priority

A where Clause object is actually a tree of Clause objects that can be combined using the And, Or and Not operators.

The main SearchService search method takes a Query object and walks down the tree of where Clause objects to create a Lucene query. The query is executed and the issues that correspond to the resulting Lucene Documents are returned in SearchResult object, sorted according to the “order” clause.

The detailed sequence of methods of how all that happens is shown in Figure 4-2.

Figure 4-2. Executing a search

Tip

You can display the original JQL query and its associated Lucene query in the JIRA log file by adding the following lines to log4j.properties file:

log4j.logger.com.atlassian.jira.issue.search.providers.LuceneSearchProvider = \
  INFO, console, filelog
log4j.additivity.com.atlassian.jira.issue.search.providers.LuceneSearchProvider 
    = false

The creation of a Lucene Query from a JQL where Clause is done by the createLuceneQuery method in the DefaultLuceneQueryBuilder class. This method walks down the tree of clauses, building up the Lucene Query as it goes and handling negation and empty clauses.

The JQL Clauses used for each specific custom field are found via the createAssociatedSearchHandler method in the CustomFieldImpl class. This method returns a SearchHandler object, which is another container of information about the searcher.

Searchers and atlassian-plugin.xml

Getting the customfield-searcher element for custom field searchers right in atlassian-plugin.xml is a confusing area, so this section covers it in some detail before we come to the examples. The JIRA documentation for the customfield-searcher plugin module type can be found at https://developer.atlassian.com/display/JIRADEV/Custom+Field+Plugin+Module, but it is somewhat sparse.

The various elements and attributes used when defining a customfield-searcher element in atlassian-plugin.xml are as follows.

key

The identifier for the searcher, unique within this plugin. This is typically something like “mycustomfield-searcher”. If this is changed later on you wont’ be able to use the searcher in a field until you edit the field and choose the searcher with the new key.

name description

The name is a short string that is displayed when a JIRA administrator chooses this searcher while creating or editing a custom field. For example, “Free Text Searcher”. The description string only appears in the list of a plugin’s modules. These can both be changed at any time.

class

This is the Java class that actually implements the searcher. This may be a class that extends one of the system searcher classes that are listed in system-customfieldtypes-plugin.xml. It could also be a plugin class that’s a totally new searcher.

resource

These elements refer to the Velocity template files that the searcher should use. The search resource controls how the search field appears in the Issue Navigator. The view resource controls how a search constraint is displayed in the Summary tab of the Issue Navigator. The label resource, if present, is used to control how statistics about the results are shown.

The resource’s files can be ones shipped with JIRA under atlassian-jira/WEB-INF/classes/templates/plugins/fields or files that are part of the searcher plugin. You can use your own searcher with JIRA’s existing Velocity templates, or you can use your own template files with extended instances of existing JIRA searcher classes—or a mixture of the two.

valid-customfieldtype

These elements control which custom fields this searcher can be used with. The package and key attributes are easy to get wrong, and then your custom field won’t have the expected searcher choices available for it.

The key attribute has to be exactly the same as the key attribute of the desired customfield-type element.

The package attribute has to be the same as the key attribute of the custom field’s top-level atlassian-plugin element. For system searchers this key is com.atlassian.jira.plugin.system.customfieldtypes. For custom fields the key is defined (by default) using the Maven variables like this:

<atlassian-plugin
      key="${project.groupId}.${project.artifactId}"
      name="${project.artifactId}"
      plugins-version="2">

For the example in Chapter 2, the valid-customfieldtype package would be com.mycompany.jira.plugins plus a period plus currency (i.e., com.mycompany.jira.plugins.currency). Note that this package attribute is a string, not a Java class or package name.

A Simple Searcher

The first searcher example is a searcher for the Currency custom field type that was created in A New Custom Field Type. This custom field type displays a number according to the local currency conventions.

The class for the custom field type extends the standard NumberCFType class so we can reuse and extend one of the searchers that are already defined for that class in system-customfieldtypes-plugin.xml.

To do this, find the customfield-type element for the NumberCFType custom field type class and note the value of its key element. In this case the value is “float”. Now search in the same file for customfield-searcher elements with valid-customfield-type elements that have a key attribute of float. This produces the two searchers that Number Field custom fields can use: NumberRangeSearch and ExactNumberSearcher. We’ll use the latter class in this example. “Exact” means that no pattern matching such as 10* is supported by the searcher.

Tip

You can tell whether an existing searcher will work with a particular custom field type if the searcher class uses a FieldIndexer class that refers to objects of the same class as the custom field type’s transport object (see Fields with Multiple Values).

In an ideal world, we would just declare a similar customfield-searcher element in the Currency plugin’s existing atlassian-plugin.xml using the same ExactNumberSearcher searcher class, and then declare the new customfield-searcher as valid for the Currency custom field type. Example 4-1 shows the new searcher in the same atlassian-plugin.xml file that was shown in Example 2-1.

Example 4-1. atlassian-plugin.xml with a searcher added

<customfield-searcher key="currencysearcher" 
                      name="Currency Searcher"
                      class="com.mycompany.jira.plugins.currency.searchers.
                          CurrencySearcher">  
    <description>
        Allow searching for a currency value.
    </description>

    <resource type="velocity" 
              name="search" 
              location="templates/com/mycompany/jira/plugins/currency/searchers/
                  search.vm"/>
    <resource type="velocity" 
              name="view" 
              location="templates/com/mycompany/jira/plugins/currency/searchers/
                  summary.vm"/>
        
    <valid-customfield-type package="com.mycompany.jira.plugins.currency" 
              key="currency-field"/> 
</customfield-searcher>

: This is the class of the custom searcher we are creating. A separate Java package name for searcher classes is generally a good idea.
: These are copies of the Velocity template files that the existing ExactNumberSearcher uses in the Number Searcher searcher.
: The search resource is what controls the HTML input field for this searcher in the Issue Navigator.
: The view resource defines what appears for the searcher in the Issue Navigator’s Summary tab.
: Make sure we get the package and key right for the custom field type that we want to search, as noted in Searchers and atlassian-plugin.xml. The custom field is defined in the same atlassian-plugin.xml file as the searcher.

The main difference from an ideal world is that the existing ExactNumberSearcher searcher class can’t be referred to directly by the customfield-searcher element because it’s not part of the plugin. To work around this we create a new class in the plugin whose sole purpose is to act as a wrapper for that searcher class. Since the searcher class is intended for the Currency field type, we’ll call it CurrencySearcher. It is shown in full in Example 4-2.

Example 4-2. CurrencySearcher.java

package com.mycompany.jira.plugins.searchers.currency;

import com.atlassian.jira.issue.customfields.converters.DoubleConverter;
import com.atlassian.jira.issue.customfields.searchers.ExactNumberSearcher;
import com.atlassian.jira.issue.customfields.searchers.transformer.CustomFieldInputHelper;
import com.atlassian.jira.jql.operand.JqlOperandResolver;
import com.atlassian.jira.util.I18nHelper;
import com.atlassian.jira.web.FieldVisibilityManager;

/**
 * A custom searcher class that simply reuses an existing searcher.
 */
public class CurrencySearcher extends ExactNumberSearcher {

    public CurrencySearcher(final FieldVisibilityManager fieldVisibilityManager,
                            final JqlOperandResolver jqlOperandResolver,
                            final DoubleConverter doubleConverter, 
                            final CustomFieldInputHelper customFieldInputHelper,
                            final I18nHelper.BeanFactory beanFactory) {
        super(fieldVisibilityManager,
              jqlOperandResolver,
              doubleConverter,
              customFieldInputHelper,
              beanFactory);
    }

}

The search.vm and summary.vm Velocity resources in atlassian-plugin.xml are copies of the template files used by the Number Field custom field with changes to add the currency symbol, just as described in Adding Velocity Template Files.

Once we have rebuilt and redeployed the plugin, we should see the new searcher as a module in the Currency plugin details in the Administration→Plugins page. We can also now change the search template used for a Currency custom field to the new Currency Searcher searcher by using the Edit link, as shown in Figure 4-3.

Figure 4-3. Using the new searcher in a custom field

After reindexing, the custom field should duly appear in the Custom Fields section of the Issue Navigator as shown in Figure 4-4, which is very similar in appearance to editing a Currency value shown in Figure 2-2. Searching for a number should find all issues that contain that number. And because we extended a standard searcher we can even use other operators such as < and >.

Figure 4-4. The new searcher in the Issue Navigator

Troubleshooting Searchers

If the searcher doesn’t appear in as a choice in a custom field’s Edit screen, check the log file for any ERROR or WARN strings associated with loading the plugin. The searcher should appear as a valid and enabled module of the plugin at the Administration→Plugins page.

If the searcher still doesn’t appear as a choice, it may be that the valid-customfieldtype element is incorrect, as discussed in the section Searchers and atlassian-plugin.xml.

If you’re developing a new custom searcher and you’re not getting the results you expect, reindex and check what’s actually in the Lucene index by using the luke tool described in Methods for a Custom Field Searcher. Increasing the logging of the LuceneSearchProvider class as described in Executing a Search can also help.

More Complex Searchers

This next example shows how we can create more complex searchers. For the first example we’re going to change what is stored in the Lucene index for the Currency custom field type. We’ll add the string Small to the Lucene index for all values less than 10, so we can easily find all the issues with small amounts in a Currency field.

The searcher class to extend for this example is AbstractInitializationCustomFieldSearcher, which is also the parent class of most of the existing searchers. The easiest way to do this is to copy the contents of the ExactNumberSearcher.java source file to a new file named CurrencyAdvancedSearcher.java and make sure that it compiles in your plugin environment. You’ll probably need to add some import statements for a few classes that are in the original searcher’s package.

Caution

All the work in a custom field searcher is done in the init method, where the four main variables listed in Methods for a Custom Field Searcher are set. What gets complicated is that each of these variables is created with a non-trivial number of other variables. Thankfully, most of the time only a few of these variables need to be changed.

The way that data is stored in the Lucene index is controlled by a FieldIndexer object in the searcher class. We can replace the NumberCustomFieldIndexer object that the original ExactNumberSearcher used with a new indexer object of class CurrencyCustomFieldIndexer. In our example this is the line in CurrencyAdvancedSearcher.java that looks like this:

      final FieldIndexer indexer = 
        new CurrencyCustomFieldIndexer(fieldVisibilityManager, field, 
            doubleConverter);

The changes in the new CurrencyCustomFieldIndexer class are shown in Example 4-3. We have modified the addDocumentFields method to add another value to the Document object in the Lucene index when the field has a value less than 10. It’s fine to have multiple entries in the Document for the same custom field id in the same issue. The entries are more like labels or tags than unique identifiers.

Example 4-3. Changes to CurrencyCustomFieldIndexer.java

package com.mycompany.jira.plugins.currency.searchers;

import com.atlassian.jira.issue.index.indexers.impl.AbstractCustomFieldIndexer;
import com.atlassian.jira.issue.Issue;
import com.atlassian.jira.issue.customfields.converters.DoubleConverter;
import com.atlassian.jira.issue.fields.CustomField;
import static com.atlassian.jira.util.dbc.Assertions.notNull;
import com.atlassian.jira.web.FieldVisibilityManager;

import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;

/**
 * A custom field indexer for Currency custom fields
 */
public class CurrencyCustomFieldIndexer extends AbstractCustomFieldIndexer {

    public final static String SMALL = "Small";    // addDocumentsFieldsSearchable, addDocumentsFieldsNotSearchable methods
    // omitted for space

    private void addDocumentFields(final Document doc, 
                                   final Issue issue, 
                                   final Field.Index indexType) {
        Object value = customField.getValue(issue);
        if (value == null) {
            return;
        }
        Double dbl = (Double)value;
        
        final String stringValue = doubleConverter.getStringForLucene(dbl);
        // This is a Lucene Field, not a JIRA CustomField
        Field field = new Field(getDocumentFieldId(), 
                                stringValue, 
                                Field.Store.YES, 
                                indexType);
        doc.add(field);

        // Add the extra information to the index
        if (dbl.doubleValue() < 10.0) {
            field = new Field(getDocumentFieldId(), 
                              SMALL,
                              Field.Store.YES, 
                              indexType);
            doc.add(field);
        }
    }

}

Now after a reindex, the Lucene index will have two values for this custom field for each issue. However because one of the values is a text string instead of a number string, we also need to change how the search input is validated.

When a value is entered in the search field in the Issue Navigator, it is validated and converted to a transport object by the custom field methods, as discussed in CustomFieldType Methods. This means that to make the string “Small” acceptable as a search term, we have to update the getSingularObjectFromString method in the CurrencyCFType custom field type of A New Custom Field Type. The change is shown in Example 4-4. Another file CurrencyIndexValueConverter.java also has to be changed slightly to handle a string (see the source code).

Example 4-4. Changes to CurrencyCFType.java

    public Double getSingularObjectFromString(String numberString)  
        throws FieldValidationException {

        // This is for the advanced currency searcher
        if (numberString.equalsIgnoreCase(CurrencyCustomFieldIndexer.SMALL)) {
            return Double.valueOf("0");
        }
    
        // code omitted for brevity
    }

The case-insensitive string “Small” is now accepted as a valid value for a Currency custom field and can be used by the searcher.

Tip

Some FieldIndexer classes store the value field’s value and also a modified value that is more useful for searching or sorting. For example, Select List options are indexed both by the raw value and also by a string with all upper-case characters converted to lower-case to make searching for an option independent of its case.

Statistical Searchers

JIRA has many gadgets and reports that allow you to summarize the number of issues by the different values in a field. For example, you can use the Issue Statistics gadget to display a histogram of the number of issues in each status. To make this possible with a custom field instead of status, the searcher for that field must implement the CustomFieldStattable interface.

The CustomFieldStattable interface has just one method: getStatisticsMapper. Most searcher classes can usually just return one of the classes that already implement the StatisticsMapper interface, or you can create a new class. The important thing is to choose a StatisticsMapper class that expects the data in the Lucene index to be the same kind of data that the searcher’s FieldIndexer added to the index. The getSorter method defined in the SortableCustomFieldSearcher interface often returns the same StatisticsMapper object as the getStatisticsMapper method.

Within a StatisticsMapper class, the getDocumentConstant method must return the value that the field was indexed with, usually the customfield_NNNNN field ID. The getValueFromLuceneField method should return an Object that is of the same type as the custom field’s singular object and one that works with the Comparator returned by the getComparator method.

We can easily modify CurrencySearcher.java and make it implement CustomFieldStattable by reusing the existing NumericFieldStatisticsMapper class, as shown in Example 4-5. Only the changes from Example 4-2 are shown, and in the source code this searcher’s file is named CurrencyStattableSearcher.java

Example 4-5. A Stattable searcher

import com.atlassian.jira.issue.customfields.statistics.CustomFieldStattable;
import com.atlassian.jira.issue.statistics.NumericFieldStatisticsMapper;
import com.atlassian.jira.issue.statistics.StatisticsMapper;

public class CurrencySearcher extends ExactNumberSearcher 
                              implements CustomFieldStattable {

    public StatisticsMapper getStatisticsMapper(final CustomField customField) {
        return new NumericFieldStatisticsMapper(customField.getId());
    }

}

However no names for the field’s values will appear in our gadgets unless we add a suitable label resource in atlassian-plugin.xml, as shown in Example 4-6. Adding new variables to the Velocity context for the label resource is not a simple matter, but the customField variable can be used to access the CustomFieldType object and its getVelocityParameters method. That’s one way to access the numberTool object so the correct currency symbol can be displayed in the label.

Example 4-6. A label resource in atlassian-plugin.xml

<resource type="velocity"
          name="label" 
          location="templates/com/mycompany/jira/plugins/currency/searchers/label.vm" />

After deploying this new searcher, edit a Currency custom field and select the new “Stattable Currency Searcher”. After reindexing, the custom field should now appear as a selection in the lists of fields that you can summarize results by in the Issue Statistics gadget. The resulting histogram is shown in Figure 4-5. If you don’t reindex you may get errors or incorrect values displayed.

Figure 4-5. Custom field searcher statistics in a gadget

Practical JIRA Plugins by Matthew B. Doar

Chapter 4. Custom Field Searchers

Overview

How Searchers Work

Methods for a Custom Field Searcher

Adding Data to the Lucene Index

Caution

Tip

Executing a Search

Tip

Searchers and atlassian-plugin.xml

A Simple Searcher

Tip

Troubleshooting Searchers

More Complex Searchers

Caution

Tip

Statistical Searchers

Further Reading

Don’t leave empty-handed

It’s yours, free.

Check it out now on O’Reilly