XSLT Cookbook

Tokenizing a String

Problem

You want to break a string into a list of tokens based on the occurrence of one or more delimiter characters.

Solution

Jeni Tennison implemented this solution (but the comments are my doing). The tokenizer returns each token as a node consisting of a <token> element text. It also defaults to character-level tokenization if the delimiter string is empty.

<xsl:template name="tokenize"> <xsl:param name="string" select="''" /> <xsl:param name="delimiters" select="' &#x9;&#xA;'" /> <xsl:choose> <!-- Nothing to do if empty string --> <xsl:when test="not($string)" /> <!-- No delimiters signals character level tokenization. --> <xsl:when test="not($delimiters)"> <xsl:call-template name="_tokenize-characters"> <xsl:with-param name="string" select="$string" /> </xsl:call-template> </xsl:when> <xsl:otherwise> <xsl:call-template name="_tokenize-delimiters"> <xsl:with-param name="string" select="$string" /> <xsl:with-param name="delimiters" select="$delimiters" /> </xsl:call-template> </xsl:otherwise> </xsl:choose> </xsl:template> <xsl:template name="_tokenize-characters"> <xsl:param name="string" /> <xsl:if test="$string"> <token><xsl:value-of select="substring($string, 1, 1)" /></token> <xsl:call-template name="_tokenize-characters"> <xsl:with-param name="string" select="substring($string, 2)" /> </xsl:call-template> </xsl:if> </xsl:template> <xsl:template name="_tokenize-delimiters"> <xsl:param name="string" /> <xsl:param name="delimiters" /> <xsl:param name="last-delimit"/> ...