book

Mastering Data Mining with Python - Find patterns hidden in your data

Name: Mastering Data Mining with Python - Find patterns hidden in your data
Author: Megan Squire
ISBN: 9781785889950

by Megan Squire

August 2016

Beginner to intermediate

268 pages

6h 34m

English

Packt Publishing

Read now

Unlock full access

Mastering Data Mining with Python – Find patterns hidden in your data
Table of Contents
Mastering Data Mining with Python – Find patterns hidden in your data
Credits
About the Author
About the Reviewers
www.PacktPub.com
eBooks, discount offers, and moreWhy subscribe?
Preface
What this book covers
What you need for this book
Who this book is for

Conventions
Reader feedback
Customer support
Downloading the example codeErrataPiracyQuestions
1. Expanding Your Data Mining Toolbox
What is data mining?
How do we do data mining?
The Fayyad et al. KDD processThe Han et al. KDD processThe CRISP-DM processThe Six Steps processWhich data mining methodology is the best?
What are the techniques used in data mining?
What techniques are we going to use in this book?
How do we set up our data mining work environment?
Summary
2. Association Rule Mining
What are frequent itemsets?The diapers and beer urban legendFrequent itemset mining basics
Towards association rules
SupportConfidenceAssociation rulesAn example with dataAdded value – fixing a flaw in the planMethods for finding frequent itemsets
A project – discovering association rules in software project tags
Summary
3. Entity Matching
What is entity matching?Merging dataMerging datasets verticallyMerging datasets horizontallyTechniques for matchingAttribute-based similarity matchingBe careful of pairwise comparisonsLeverage rare valuesMethods for matching attributesRange-based or distance from targetString edit distanceHamming distanceLevenshtein distanceSoundexLeveraging disjoint setsContext-based similarity matchingMachine learning-based entity matchingEvaluation of entity matching techniquesEfficiency – how long does it take to do the matching?Effectiveness – how accurate are the matches that we generate?Usefulness – how practical is the matching procedure to use?
Entity matching project
Difficulties with matching software projectsTwo examplesMatching on project namesMatching on people namesMatching on URLsMatching on topics and description keywordsThe datasetThe codeThe resultsHow many entity matches did we find?How good are the pairs we found?
Summary
4. Network Analysis
What is a network?
Measuring a network
Degree of a networkDiameter of a networkWalks, paths, and trails in a networkComponents of a networkCentrality of a networkCloseness centralityDegree centralityBetweenness centralityOther measures of centrality
Representing graph data
Adjacency matrixEdge lists and adjacency listsDifferences between graph data structuresImporting data into a graph structureAdjacency list formatEdge list formatGEXF and GraphMLGDFPython pickleJSONJSON node and link seriesJSON treesPajek format
A real project
Exploring the dataGenerating the network filesUnderstanding our data as a networkGenerating simple network metricsPlaying with the parameters of a networkAnalyzing subgraphsAnalyzing cliques and centrality in the subgraphsLooking for change over time
Summary
5. Sentiment Analysis in Text
What is sentiment analysis?
The basics of sentiment analysis
The structure of an opinionDocument-level and sentence-level analysisImportant features of opinions
Sentiment analysis algorithms
General-purpose data collectionsHu and Liu's sentiment analysis lexiconSentiWordNetVader sentiment
Sentiment mining application
Motivating the projectData preparationData analysis of chat messagesData analysis of e-mail messages
Summary
6. Named Entity Recognition in Text
Why look for named entities?
Techniques for named entity recognition
Tagging parts of speechClasses of named entities
Building and evaluating NER systems
NER and partial matchesHandling partial matches
Named entity recognition project
A simple NER toolApache Board meeting minutesDjango IRC chatGnuIRC summariesLKML e-mails
Summary
7. Automatic Text Summarization
What is automatic text summarization?
Tools for text summarization
Naive text summarization using NLTKText summarization using GensimText summarization using SumySumy's Luhn summarizerSumy's TextRank summarizerSumy's LSA summarizerSumy's Edmundson summarizer
Summary
8. Topic Modeling in Text
What is topic modeling?
Latent Dirichlet Allocation
Gensim for topic modeling
Understanding Gensim LDA topicsUnderstanding Gensim LDA passesApplying a Gensim LDA model to new documentsSerializing Gensim LDA objectsSerializing a dictionarySerializing a corpusSerializing a model
Gensim LDA for a larger project
Summary
9. Mining for Data Anomalies
What are data anomalies?Missing dataLocating missing dataZero valuesFixing missing dataIgnore the problem rowsFix the problem manuallyUse a fabricated valueUse a central measureUse Last Observation Carried ForwardUse a similar valueUse the most likely valueData errorsTruncated fieldsData type and character set errorsLogic or semantic errorsOutliersVisual mining for outliersStatistical detection of outliersDetecting outliers with modified z-scoresDetecting outliers by combining statistics and visual miningDetecting outliers with machine learning
Summary
Index

Content preview from Mastering Data Mining with Python - Find patterns hidden in your data

Latent Dirichlet Allocation

The most common technique currently in use for topic modeling of text, and the one that the Facebook researchers used in their 2013 paper, is called Latent Dirichlet Allocation (LDA).

Tip

Many people wonder how to pronounce Dirichlet in English. The most common pronunciation I have heard is DEER-uh-shlay, and I have also heard DEER-uh-klay a few times.

LDA was first proposed for text topic extraction by David Blei, Andrew Ng, and Michael Jordan in a 2003 paper entitled simply Latent Direchlet Allocation, available from the Journal of Machine Learning Research at http://www.jmlr.org/papers/volume3/blei03a/blei03a.pdf. Blei also wrote a good follow-up article in 2012 for the Communications of the ACM about LDA and some ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Python for Data Mining Quick Syntax Reference

Publisher Resources

ISBN: 9781785889950

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Mastering Data Mining with Python - Find patterns hidden in your data

by Megan Squire

Latent Dirichlet Allocation

Tip

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.