book

Advanced Perl Programming, 2nd Edition

by Simon Cozens

June 2005

Intermediate to advanced

298 pages

7h 28m

English

O'Reilly Media, Inc.

Read now

Unlock full access

AudienceContentsConventions Used in This BookUsing Code ExamplesWe’d Like to Hear from YouSafari® EnabledAcknowledgments
1.1. Introspection1.1.1. Preparatory Work: Fun with Globs1.1.1.1. Aliasing1.1.1.2. Accessing parts of a glob1.1.1.3. Creating subroutines with glob assignment1.1.2. AUTOLOAD1.1.3. CORE and CORE::GLOBAL1.1.4. Case Study: Hook::LexWrap1.1.5. Introspection with B1.2. Messing with the Class Model1.2.1. UNIVERSAL1.2.2. Dynamic Method Resolution1.2.3. Case Study: Singleton Methods1.3. Unexpected Code1.3.1. Overloading1.3.1.1. Simple operator overloading1.3.1.2. Other operator overloading1.3.1.3. Non-operator overloading1.3.2. Time Shifting1.3.2.1. Doing things now with eval/BEGIN1.3.2.2. Doing things later with CHECK1.3.2.3. Doing things at the end with DESTROY1.3.2.4. Case study: Acme::Dot1.4. Conclusion
2.1. Parse::RecDescent Grammars2.1.1. Simple Parsing with Parse::RecDescent2.1.1.1. Types of match2.1.1.2. Actions2.1.2. Debugging2.1.3. More Difficult Parsing2.1.3.1. Commit, reject, and error2.1.3.2. Syntactic whitespace2.1.3.3. Automating the process2.1.3.4. And much more...2.1.4. Some Examples2.1.4.1. Parsing iCalendar data2.1.4.2. .procmailrc2.2. Parse::Yapp2.3. Other Parsing Techniques2.3.1. HTML::Parser2.3.2. XML Parsing2.3.3. And Everything Else...2.4. Conclusion
3.1. Formats and Text::Autoformat3.1.1. Text::Autoformat3.2. Text::Template3.2.1. Loops, Arrays, and Hashes3.2.2. Security and Error Checking3.2.3. Text::Template Tricks3.3. HTML::Template3.3.1. Variables and Conditions3.3.2. Loops3.3.3. RSS Aggregation3.4. HTML::Mason3.4.1. Basic Components3.4.2. Basic Dynamism3.4.3. Perl Blocks3.4.4. Our RSS Aggregator3.5. Template Toolkit3.5.1. Filters3.5.2. Plugins3.5.3. Components and Macros3.5.4. The RSS Aggregator3.6. AxKit3.7. Conclusion
4.1. Beyond Flat Files4.2. Object Serialization4.2.1. Our Schema and Classes4.2.2. Dumping Data4.2.3. Storing and Retrieving Data4.2.4. The Pruning Problem4.2.5. Multilevel DBMs4.2.6. Pixie4.3. Object Databases4.3.1. Object Database Pitfalls4.3.2. Tangram4.3.2.1. Tangram CRUD: create, read, update, delete4.4. Database Abstraction4.4.1. Trivial Mapping4.4.2. Class::DBI4.4.2.1. Relationships4.4.2.2. Class::DBI extensions4.4.3. Other Contenders4.5. Practical Uses in Web Applications4.5.1. Class::DBI and the Template Toolkit4.5.2. Maypole4.5.3. Other Application Frameworks4.6. Conclusion
5.1. Perl and Natural Languages5.2. Handling English Text5.2.1. Pluralizations and Inflections5.2.2. Converting Words to Numbers5.3. Modules for Parsing English5.3.1. Splitting Up Text5.3.2. Stemming and Stopwording5.4. Categorization and Extraction5.4.1. Bayesian Analysis5.4.2. Keyword Extraction and Summary5.4.2.1. Keyword extraction5.4.3. Extracting Names and Places5.5. Conclusion
6.1. Terminology6.2. What Is Unicode?6.2.1. What Is UCS?6.2.2. What is the Unicode Consortium?6.2.3. Why Should I Care?6.3. Unicode Transformation Formats6.3.1. UCS-26.3.2. UTF-86.3.3. UTF-16BE6.3.4. UTF-16LE6.3.5. UCS-46.3.6. UTF-326.3.7. UTF-EBCDIC6.3.8. UTF-76.4. Handling UTF-8 Data6.4.1. Entering Unicode Characters6.4.2. Unicode Regular Expressions6.5. Encode6.5.1. The PerlIO Trick6.5.2. The Gory Details6.6. Unicode for XS Authors6.6.1. Traversing Strings6.6.2. Encoding Strings6.7. Conclusion
7.1. Programming in an Event-Driven Environment7.1.1. Hello, POE7.1.2. Hello, Again, POE!7.1.3. Wheels7.1.4. A Port Forwarder7.2. Top-Level Pieces: Components7.2.1. Medium-Level Components7.2.2. A POE Web Server7.2.3. Highest-Level Components7.3. Conclusion
8.1. Test::Simple8.2. Test::More8.2.1. Skips and Todos8.2.2. Automated Tests8.3. Test::Harness8.4. Test::Builder8.5. Test::Builder::Tester8.6. Keeping Tests and Code Together8.7. Unit Tests8.7.1. Test::Class8.7.2. Test::MockObject8.7.3. Testing Apache, DBI, and Other Complex Environments8.8. Conclusion
9.1. Simple Inline::C9.1.1. Taking and Giving9.1.2. C is not Always a Win9.2. More Complex Tasks with Inline::C9.2.1. Dealing with Perl’s Internal Values9.2.2. Handling the Stack9.2.3. Handling More Complex Perl Types9.2.3.1. References9.2.3.2. Arrays9.2.3.3. Hashes9.2.4. Wrapping C Libraries9.2.5. Debugging Inline Extensions9.2.6. Packaging Inline Modules9.3. Inline:: Everything Else9.3.1. Inline::Python9.3.2. Inline::Ruby9.3.3. Inline::CPR9.3.4. Inline::Struct9.3.5. Miscellaneous Other Inlines9.4. Conclusion

10.1. Obfuscation10.2. Just Another Perl Hacker10.3. Perl Golf10.4. Perl Poetry10.5. Acme::*10.6. Conclusion

Content preview from Advanced Perl Programming, 2nd Edition

Chapter 5. Natural Language Tools

Sean Burke, author of Perl and LWP and a professional linguist, once described artificial intelligence as the study of programming situations where you either don’t know what you want or don’t know how to get it.

Natural-language processing, or NLP, is the application of AI techniques to answer questions about text written in a human language: what does it mean, what other documents is it like, and so on. As Perl is often described as a text-processing language, it shouldn’t be much of a surprise to find that there are a great many modules and techniques available in Perl for carrying out NLP-related tasks.

But as we’ve seen so far in this book, the real strength of Perl is not in the ease with which we can program particular techniques, but that so many of the techniques we need—techniques to break texts into sentences and words, to correctly strip the endings off inflected words, to put the right endings back on again, and so on—have already been implemented and placed on CPAN. So in this chapter we’re going to take a tour of the natural language section of CPAN, and see how we can use its modules to slice and dice any language text we need to deal with.

Perl and Natural Languages

There’s an especially good reason why Perl is used for handling natural language problems—Perl was created with natural languages in mind. In fact, Perl’s creator, Larry Wall, has a joint degree in natural and artificial languages and sees Perl as influenced by both branches ...