book

IBM Content Analytics Version 2.2: Discovering Actionable Insight from Your Content

by Wei-Dong (Jackie) Zhu, Asako Iwai, Todd Leyba, Josemina Magdalen, Kristin McNeil, Tetsuya Nasukawa, Nitaben (Nita) Patel, Kei Sugano

May 2011

Intermediate to advanced

698 pages

14h 57m

English

IBM Redbooks

Read now

Unlock full access

Front cover
Notices
Trademarks
Preface
The team who wrote this bookNow you can become a published author, too!Comments welcomeStay connected to IBM Redbooks
Summary of changes
May 2011, Second Edition
Chapter 1. Overview of IBM Content Analytics
1.1 Business need and the Content Analytics solution1.1.1 Business need and problem statement1.1.2 The Content Analytics solution1.2 History, changes, and whatâ€™s new1.2.1 Product history1.2.2 Product changes1.2.3 Whatâ€™s new in Content Analytics Version 2.21.3 Important concepts and terminology1.3.1 Unstructured and structured content1.3.2 Text analytics1.3.3 Search, discovery, and data mining1.3.4 Collections1.3.5 Facets1.3.6 Frequency1.3.7 Correlation1.3.8 Deviation1.4 Content Analytics architecture1.4.1 Main components1.4.2 Data flow1.4.3 Scalability1.4.4 Security
Chapter 2. Application design and preparation
2.1 Use-case scenarios2.1.1 Call center2.1.2 Insurance fraud2.1.3 Quality assurance2.1.4 Content assessment2.2 Data considerations2.2.1 Content Analytics data model2.2.2 Structured and unstructured sources2.2.3 Multiple data sources2.2.4 Date-sensitive data2.2.5 Extracting information from textual data2.2.6 The number of collections to use2.3 Design guide for building a text analytics collection2.3.1 Building a text analytics collection2.3.2 A walk through the building process2.3.3 Planning for iteration2.4 Programming interfaces2.4.1 Search and Index API2.4.2 REST API
Chapter 3. Understanding content analysis
3.1 Basic concepts of Content Analytics3.1.1 Manual versus automated analysis3.1.2 Frequency versus deviation3.1.3 Precision versus recall3.2 Typical cycle of analysis with Content Analytics3.2.1 Setting the objectives of the analysis3.2.2 Gathering data3.2.3 Analyzing data3.2.4 Taking action based on the analysis3.2.5 Validating the effect3.3 Successful use cases3.3.1 Voice of customer3.3.2 Analysis of other data3.4 Summary
Chapter 4. Installing and configuring IBM Content Analytics
4.1 Installing Content Analytics4.1.1 Process overview4.1.2 Confirming the system requirements and supported data sources4.1.3 Determining the installation server type and procedure consideration4.1.4 Parameters used during the installation4.1.5 Installing the agent server4.1.6 Installing Content Analytics on a single server4.1.7 Running the First Steps program to verify the installation4.1.8 Starting the Text Analytics tutorial4.2 Administering Content Analytics4.2.1 Starting the system4.2.2 Accessing the administration console4.2.3 Stopping the server4.3 Configuring a text analytics collection4.3.1 Designing a sample collection4.3.2 Creating a text analytics collection4.3.3 Defining and configuring a crawler4.3.4 Building an index in the text analytics collection4.4 Verifying that the collection is available4.4.1 Starting the search server for the text analytics collection4.4.2 Accessing the collection with the text miner application4.5 Deploying the configuration4.5.1 Using the esadmin export and import commands4.5.2 Using the esbackup and esrestore commands4.5.3 Usage guidelines for these commands
Chapter 5. Text miner application: Basic features
5.1 Overview of the text miner application5.1.1 Accessing the text miner application5.1.2 Application window layout and functional overview5.1.3 Selecting a collection for analysis5.1.4 Changing the default behavior by using preferences5.2 Search and discovery features5.2.1 Limiting the scope of your analysis using facets5.2.2 Limiting the scope of your analysis using search operators5.2.3 Limiting the scope of your analysis using dates5.2.4 Query syntax5.2.5 Type ahead5.2.6 Saved searches5.2.7 Advanced search5.3 Query tree5.3.1 Accessing the query tree5.3.2 Understanding the query tree5.3.3 Query tree examples5.3.4 Editing the query tree5.4 Query builder5.4.1 Accessing the query builder5.4.2 Features of the Query Builder window5.4.3 Using the query builder5.4.4 Preferred practice for using the query builder and the query tree5.5 Rule-based categories with a query5.5.1 Enabling the rule-based categories feature5.5.2 Configuring rules for rule-based categories5.5.3 Configuring rule-based categories5.5.4 Adding the current query as a category rule5.6 Common view features5.7 Document flagging5.7.1 Configuring document flags5.7.2 Setting document flags5.7.3 Viewing the document values of a flag facet
Chapter 6. Text miner application: Views
6.1 Views6.2 Documents view6.2.1 Understanding the Documents view6.2.2 Viewing the document contents and facets6.2.3 When to use the Documents view6.3 Facets view6.3.1 Understanding the Facets view6.3.2 Using the Facets view6.4 Time Series view6.4.1 Features in the Time Series view6.4.2 Understanding the Time Series view6.4.3 Using the Time Series view6.5 Trends view6.5.1 Features in the Trends view6.5.2 Sort criteria6.5.3 Understanding the Trends view6.5.4 When to use the Trends view6.6 Deviations view6.6.1 Features in the Deviations view6.6.2 Understanding the Deviations view6.6.3 Using the Deviations view6.7 Facet Pairs view6.7.1 Table view6.7.2 Grid view6.7.3 Birdâ€™s eye view6.7.4 Understanding the Facet Pairs view with correlation values6.7.5 Using the Facet Pairs view6.8 Connections view6.8.1 Features in the Connections view6.8.2 Understanding the Connections view6.8.3 When to use the Connections view6.9 Dashboard view6.9.1 Configuring the Dashboard6.9.2 Viewing the Dashboard6.9.3 Working with the Dashboard6.9.4 Saving Dashboard charts as images

Chapter 7. Performing content analysis
7.1 Discovering actionable insight with the text miner application7.1.1 The sample data7.1.2 Insights without customization7.1.3 Considerations about what you want to discover from the data7.2 Content analysis scenarios7.2.1 Scenario 1: Using a custom dictionary to discoverpackage-related calls7.2.2 Scenario 2: Using custom text analysis rules to discover trouble-related calls7.2.3 Scenario 3: Discovering the cause of increasing calls7.2.4 Conclusion7.3 Configuring the Dictionary Lookup annotator7.3.1 When to use the Dictionary Lookup annotator7.3.2 Configuring custom user dictionaries7.3.3 Migrating the Content Analyzer dictionaries7.3.4 Validation and maintenance7.4 Configuring the Pattern Matcher annotator7.4.1 When to use the Pattern Matcher annotator7.4.2 Configuring custom text analysis rules7.4.3 Migrating the Content Analyzer rules7.4.4 Designing the custom text analysis rules7.4.5 Validation and maintenance7.5 Preferred practices
Chapter 8. Discovering insight with terms of interest and document clustering
8.1 The power of dictionary-driven analytics8.1.1 Multiple viewpoints for analyzing the same data8.2 Terms of interest8.2.1 Basic algorithm for identifying terms of interest8.2.2 Limitations in using automatic identification of terms of interest8.2.3 Preferred use of terms of interest identified automatically8.2.4 Efficient and effective creation of dictionary8.3 Document clustering8.3.1 Setting up document cluster8.3.2 Creating a cluster proposal8.3.3 Refining the cluster results8.3.4 Deploying clusters to a category8.3.5 Working with the cluster results8.3.6 Creating and deploying the clustering resource8.3.7 Preferred practices
Chapter 9. Content analysis with IBM Classification Module
9.1 The Classification Module annotator9.1.1 When to use the Classification Module annotator9.1.2 The Classification Module technology9.2 Fine-tuning your analysis with the Classification Module annotator9.2.1 Building your collection9.2.2 Refining the analysis9.2.3 Using a conceptual search for advanced content discovery9.3 Creating and deploying the Classification Module resource9.3.1 Starting the Classification Module server9.3.2 Creating and training the knowledge bases9.3.3 Creating a decision plan9.3.4 Deploying the knowledge base and decision plan9.3.5 Configuring the Classification Module annotator9.4 Validation and maintenance of the Classification Module annotator9.4.1 Using the Classification Module sample programs9.4.2 Classification Module annotator validation techniques9.5 Preferred practices
Chapter 10. Importing CSV files, exporting data, and performing deep inspection
10.1 Importing CSV files10.2 Overview of exporting documents and data10.2.1 Crawled documents10.2.2 Analyzed documents10.2.3 Search result documents10.2.4 Exported data manifest10.3 Location and format of the exported data10.3.1 Location of the exported data10.3.2 Metadata format10.3.3 Binary content format10.3.4 Common Analysis Structure format10.3.5 Extracted text format10.4 Common configuration of the export feature10.4.1 Document URI pattern10.4.2 Exporting XML attributes and preserving file extensions10.4.3 Adding exported documents to the index10.4.4 Exporting information about deleted documents10.4.5 Scheduling10.5 Monitoring export requests10.6 Enabling export and sample configurations10.6.1 Exporting crawled documents to a file system for Content Collector10.6.2 Exporting analyzed documents to a relational database10.6.3 Exporting search result documents to the file system for Classification Module10.6.4 Exporting search result documents to CSV files10.7 Deep inspection10.7.1 Location and format of the exported data10.7.2 Common configuration10.7.3 Enabling deep inspection10.7.4 Generating deep inspection reports10.7.5 Optional: Scheduling a deep inspection run10.7.6 Monitoring the deep inspection requests10.7.7 Validating the deep inspection reports generation10.8 Creating and deploying a custom plug-in
Chapter 11. Configuring annotators
11.1 Document processing pipeline and the annotators11.1.1 UIMA document processing pipeline11.1.2 Language Identification annotator11.1.3 Linguistic Analysis annotator11.1.4 Named Entity Recognition annotator11.1.5 Dictionary Lookup and Pattern Matcher annotators11.1.6 Classification Module annotator11.2 Custom annotators11.2.1 LanguageWare Resource Workbench11.2.2 Creating custom annotators using the Apache UIMA SDK11.3 Validation11.3.1 Real-time NLP11.3.2 Advanced techniques11.3.3 Summary of validation techniques
Chapter 12. IBM Content Assessment scenario
12.1 Content Assessment offering12.1.1 Concepts and terminology12.2 Overview of Content Assessment12.2.1 Content decommissioning scenario12.2.2 Dynamically analyzing and collecting your content12.3 Content Assessment workflow12.3.1 Decommissioning content12.3.2 Performing dynamic analysis12.3.3 Preserving and using business data12.4 Records management and email archiving12.5 Preferred practices12.6 Summary
Chapter 13. Integrating Cognos Business Intelligence
13.1 Initial setup13.1.1 Running the esrepcog command13.1.2 Configuring default application user roles13.1.3 Configuring database connectivity13.2 Generating Cognos BI reports13.3 Creating custom Cognos 8 BI reports13.3.1 Configuring export options13.3.2 Exporting search results13.3.3 Loading the exported data model into Cognos
Chapter 14. Customizing and extending the text miner application
14.1 Customizing the text miner application14.1.1 Analytics Customizer14.1.2 Modifying the URI link in the Documents view14.2 Reasons for extending the text miner application14.3 Sample plug-ins for text miner views14.4 Customizing the sample text miner plug-in14.4.1 Changing the view tab title14.4.2 Customizing the plug-in template HTML file14.4.3 Customizing the javascript widget14.4.4 Updating the style sheet for the plug-in14.5 Testing the customized plug-in
Chapter 15. Performance tips
15.1 General performance guidelines15.1.1 Factors that influence the performance of the system15.1.2 Variables15.2 Tuning the crawler component15.2.1 Increasing active crawler threads15.2.2 Setting the maximum heap size15.3 Tuning the document processor15.3.1 Setting the number of document processor threads15.3.2 Increasing the maximum heap size15.4 Tuning the indexer15.4.1 Setting the number of indexer threads15.4.2 Specifying the taxonomy cache type15.4.3 Increasing the maximum heap size15.4.4 Increasing the buffer size15.4.5 Increasing the index commit interval15.5 Enhancing the search performance15.5.1 Increasing the search result cache entries15.5.2 Enabling the optional facet index15.5.3 Increasing the maximum heap size15.5.4 Setting the number of threads for rebuilding optional facet index15.6 Scalability15.7 Monitoring the system15.7.1 General guideline for monitoring the system15.7.2 Using the esadmin command utility15.7.3 General guidelines for monitoring the operating system
Chapter 16. Hints and tips for troubleshooting
16.1 Overview of troubleshooting16.2 General troubleshooting guidelines16.2.1 Understanding the problem16.2.2 Understanding the environment16.3 Working with the logs in Content Analytics16.3.1 The location of the logs16.3.2 Understanding the log contents16.3.3 The first log to examine16.4 Installation and administration-related troubleshooting16.5 Text miner application-related troubleshooting16.6 Data processing flow-related troubleshooting16.7 Export-related troubleshooting16.8 Classification Module server-related troubleshooting tips16.9 Reporting a problem to the IBM Software Support16.10 Advanced troubleshooting topics16.10.1 Changing the log level of the system and collection logs16.10.2 Generating a javacore or a heapdump for a Java session
Appendix A. Security in IBM Content Analytics
The security concept in Content AnalyticsEnabling login security in the embedded application server (Jetty)Configuring application user rolesLimiting user access to the text analytics collection
Related publications
IBM RedbooksOther publicationsOnline resourcesHelp from IBM
Back cover

Content preview from IBM Content Analytics Version 2.2: Discovering Actionable Insight from Your Content

Application design and preparation

To achieve the most from your textual content, it is important to spend some time thinking about the overall design of your text analytics application and your expectations of the resulting analysis. You must do this step before attempting to work with IBM Content Analytics. Skipping this step might result in data mining results that you did not intend to have or might not find useful. A carefully designed text analytics collection and corresponding text miner application can unlock the value in your text and ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

IBM Watson Content Analytics: Discovering Actionable Insight from Your Content

Publisher Resources

ISBN: 0738435287Purchase book

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

IBM Content Analytics Version 2.2: Discovering Actionable Insight from Your Content

by Wei-Dong (Jackie) Zhu, Asako Iwai, Todd Leyba, Josemina Magdalen, Kristin McNeil, Tetsuya Nasukawa, Nitaben (Nita) Patel, Kei Sugano

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.