10.6. Implementing a Better Tokenizer
Problem
You need a tokenizer—also referred to as a lexer—that can split up a string based on a well-defined set of characters.
Solution
Using the Split method of the Regex class, you can use a regular expression to indicate the types of tokens and separators that you are interested in gathering. This technique works especially well with equations, since the tokens of an equation are well defined. For example, the code:
using System;
using System.Text.RegularExpressions;
public static string[] Tokenize(string equation)
{
Regex re = new Regex(@"([\+\–\*\(\)\^\\])");
return (re.Split(equation));
}will divide up a string according to the regular expression specified in the Regex constructor. In other words, the string passed in to the Tokenize method will be divided up based on the delimiters +, –, *, (, ), ^, and \. The following method will call the Tokenize method to tokenize the equation (y – 3)*(3111*x^21 + x + 320):
public static void TestTokenize()
{
foreach(string token in Tokenize("(y – 3)*(3111*x^21 + x + 320)"))
Console.WriteLine("String token = " + token.Trim());
}which displays the following output:
string token = String token = ( String token = y String token = - String token = 3 String token = ) String token = * String token = ( String token = 3111 String token = * String token = x String token = ^ String token = 21 String token = + String token = x String token = + String token = 320 String token = ) String token =
Notice that each individual ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access