Natural Language Processing with Python

Appendix A. Afterword: The Language Challenge

Natural language throws up some interesting computational challenges. We’ve explored many of these in the preceding chapters, including tokenization, tagging, classification, information extraction, and building syntactic and semantic representations. You should now be equipped to work with large datasets, to create robust models of linguistic phenomena, and to extend them into components for practical language technologies. We hope that the Natural Language Toolkit (NLTK) has served to open up the exciting endeavor of practical natural language processing to a broader audience than before.

In spite of all that has come before, language presents us with far more than a temporary challenge for computation. Consider the following sentences which attest to the riches of language:

1. Overhead the day drives level and grey, hiding the sun by a flight of grey spears. (William Faulkner, As I Lay Dying, 1935)

2. When using the toaster please ensure that the exhaust fan is turned on. (sign in dormitory kitchen)

3. Amiodarone weakly inhibited CYP2C9, CYP2D6, and CYP3A4-mediated activities with Ki values of 45.1-271.6 μM (Medline, PMID: 10718780)

4. Iraqi Head Seeks Arms (spoof news headline)

5. The earnest prayer of a righteous man has great power and wonderful results. (James 5:16b)

6. Twas brillig, and the slithy toves did gyre and gimble in the wabe (Lewis Carroll, Jabberwocky, 1872)

7. There are two ways to do this, AFAIK :smile: (Internet discussion archive)

Other evidence ...