November 2018
Beginner
304 pages
7h 35m
English
Strings can be combined (joined or concatenated) and separated (split) quite easily. Tokenization is the process of splitting something up into individual tokens; in this case, a sentence is split into individual words. When a web page is parsed by a browser, the HTML, JavaScript, and any other code in the page is tokenized and identified as a keyword, operator, variable, and so on. The browser then uses this information to display the web page correctly, or at least as well as it can.
Python does much the same thing. The Python interpreter tokenizes the source code and identifies the parts that are part of the actual programming language and the parts that are data. The individual tokens are separated by ...
Read now
Unlock full access