
This is the Title of the Book, eMatter Edition
Copyright © 2007 O’Reilly & Associates, Inc. All rights reserved.
574
|
Chapter 10: Regular Expressions
will divide up a string according to the regular expression specified in the Regex con-
structor. In other words, the string passed in to the
Tokenize method will be divided
up based on the delimiters
+, -, *, (, ), ^, and \. The following method will call the
Tokenize method to tokenize the equation (y - 3)(3111*x^21 + x + 320):
public void TestTokenize( )
{
foreach(string token in Tokenize("(y - 3)(3111*x^21 + x + 320)"))
Console.WriteLine("String token = " + token.Trim( ));
}
which displays the following output:
String token =
String token = (
String token = y
String token = -
String token = 3
String token = )
String token =
String token = (
String token = 3111
String token = *
String token = x
String token = ^
String token = 21
String token = +
String token = x
String token = +
String token = 320
String token = )
String token =
Notice that each individual operator, parenthesis, and number has been broken out
into its own separate token.
Discussion
The tokenizer created in Recipe 2.6 would be useful in specific controlled circum-
stances. However, in real-world projects, you do not always have the luxury of being
able to control the set of inputs to your code. By making use of regular expressions,
you can take the original tokenizer and make it flexible enough to