Java: Data Science Made Easy

Java core tokenizers

StringTokenizer was the first and most basic tokenizer and has been available since Java 1. It is not recommended for use in new development as the String class's split method is considered more efficient. While it does provide a speed advantage for files with narrowly defined and set delimiters, it is less flexible than other tokenizer options. The following is a simple implementation of the StringTokenizer class that splits a string on spaces:

StringTokenizer tokenizer = new StringTokenizer(dirtyText," "); while(tokenizer.hasMoreTokens()){   out.print(tokenizer.nextToken() + " "); } 

When we set the  dirtyText variable to hold our text from Moby Dick, shown previously, we get the following truncated output:

Call me ...