O'Reilly logo

C# Cookbook by Jay Hilyard, Stephen Teilhet

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

2.6. The Poor Man’s Tokenizer

Problem

You need a quick method of breaking up a string into a series of discrete tokens or words.

Solution

Use the Split instance method of the string class. For example:

string equation = "1 + 2 - 4 * 5";
string[] equationTokens = equation.Split(new char[1]{' '});

foreach (string Tok in equationTokens)
   Console.WriteLine(Tok);

This code produces the following output:

1
+
2
-
4
*
5

The Split method may also be used to separate people’s first, middle, and last names. For example:

string fullName1 = "John Doe";
string fullName2 = "Doe,John";
string fullName3 = "John Q. Doe";

string[] nameTokens1 = fullName1.Split(new char[3]{' ', ',', '.'});
string[] nameTokens2 = fullName2.Split(new char[3]{' ', ',', '.'});
string[] nameTokens3 = fullName3.Split(new char[3]{' ', ',', '.'});

foreach (string tok in nameTokens1)
{
   Console.WriteLine(tok);
}
Console.WriteLine("");

foreach (string tok in nameTokens2)
{
   Console.WriteLine(tok);
}
Console.WriteLine("");

foreach (string tok in nameTokens3)
{
   Console.WriteLine(tok);
}

This code produces the following output:

John
Doe

Doe
John

John
Q

Doe

Notice that a blank is inserted between the '.' and the space delimiters of the fullName3 name; this is correct behavior. If you did not want to process this space in your code, you can choose to ignore it.

Discussion

If you have a consistent string whose parts, or tokens, are separated by well-defined characters, the Split function can tokenize the string. Tokenizing a string consists ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required