book

Python: Real-World Data Science

Name: Python: Real-World Data Science
ISBN: 9781786465160

by Dusty Phillips, Fabrizio Romano, Phuong Vo.T.H, Martin Czygan, Robert Layton, Sebastian Raschka

June 2016

Beginner to intermediate

1255 pages

29h 1m

English

Packt Publishing

Read now

Unlock full access

Python: Real-World Data Science
Table of Contents
Python: Real-World Data Science
Meet Your Course Guide
What's so cool about Data Science?
Course Structure
Course Journey
The Course Roadmap and Timeline
1. Course Module 1: Python Fundamentals
1. Introduction and First Steps – Take a Deep Breath
A proper introduction

Enter the Python
About Python
PortabilityCoherenceDeveloper productivityAn extensive librarySoftware qualitySoftware integrationSatisfaction and enjoyment
What are the drawbacks?
Who is using Python today?
Setting up the environment
Python 2 versus Python 3 – the great debate
What you need for this course
Installing PythonInstalling IPythonInstalling additional packages
How you can run a Python program
Running Python scriptsRunning the Python interactive shellRunning Python as a serviceRunning Python as a GUI application
How is Python code organized
How do we use modules and packages
Python's execution model
Names and namespacesScopes
Guidelines on how to write good code
The Python culture
A note on the IDEs
2. Object-oriented Design
Introducing object-oriented
Objects and classes
Specifying attributes and behaviors
Data describes objectsBehaviors are actions
Hiding details and creating the public interface
Composition
Inheritance
Inheritance provides abstractionMultiple inheritance
Case study
3. Objects in Python
Creating Python classesAdding attributesMaking it do somethingTalking to yourselfMore argumentsInitializing the objectExplaining yourself
Modules and packages
Organizing the modulesAbsolute importsRelative imports
Organizing module contents
Who can access my data?
Third-party libraries
Case study
4. When Objects Are Alike
Basic inheritanceExtending built-insOverriding and super
Multiple inheritance
The diamond problemDifferent sets of arguments
Polymorphism
Abstract base classes
Using an abstract base classCreating an abstract base classDemystifying the magic
Case study
5. Expecting the Unexpected
Raising exceptionsRaising an exceptionThe effects of an exceptionHandling exceptionsThe exception hierarchyDefining our own exceptions
Case study
6. When to Use Object-oriented Programming
Treat objects as objects
Adding behavior to class data with properties
Properties in detailDecorators – another way to create propertiesDeciding when to use properties
Manager objects
Removing duplicate codeIn practice
Case study
7. Python Data Structures
Empty objects
Tuples and named tuples
Named tuples
Dictionaries
Dictionary use casesUsing defaultdictCounter
Lists
Sorting lists
Sets
Extending built-ins
Queues
FIFO queuesLIFO queuesPriority queues
Case study
8. Python Object-oriented Shortcuts
Python built-in functionsThe len() functionReversedEnumerateFile I/OPlacing it in context
An alternative to method overloading
Default argumentsVariable argument listsUnpacking arguments
Functions are objects too
Using functions as attributesCallable objects
Case study
9. Strings and Serialization
StringsString manipulationString formattingEscaping bracesKeyword argumentsContainer lookupsObject lookupsMaking it look rightStrings are UnicodeConverting bytes to textConverting text to bytesMutable byte strings
Regular expressions
Matching patternsMatching a selection of charactersEscaping charactersMatching multiple charactersGrouping patterns togetherGetting information from regular expressionsMaking repeated regular expressions efficient
Serializing objects
Customizing picklesSerializing web objects
Case study
10. The Iterator Pattern
Design patterns in brief
Iterators
The iterator protocol
Comprehensions
List comprehensionsSet and dictionary comprehensionsGenerator expressions
Generators
Yield items from another iterable
Coroutines
Back to log parsingClosing coroutines and throwing exceptionsThe relationship between coroutines, generators, and functions
Case study
11. Python Design Patterns I
The decorator patternA decorator exampleDecorators in Python
The observer pattern
An observer example
The strategy pattern
A strategy exampleStrategy in Python
The state pattern
A state exampleState versus strategyState transition as coroutines
The singleton pattern
Singleton implementation
The template pattern
A template example
12. Python Design Patterns II
The adapter pattern
The facade pattern
The flyweight pattern
The command pattern
The abstract factory pattern
The composite pattern
13. Testing Object-oriented Programs
Why test?Test-driven development
Unit testing
Assertion methodsReducing boilerplate and cleaning upOrganizing and running testsIgnoring broken tests
Testing with py.test
One way to do setup and cleanupA completely different way to set up variablesSkipping tests with py.test
Imitating expensive objects
How much testing is enough?
Case study
Implementing it
14. Concurrency
ThreadsThe many problems with threadsShared memoryThe global interpreter lockThread overhead
Multiprocessing
Multiprocessing poolsQueuesThe problems with multiprocessing
Futures
AsyncIO
AsyncIO in actionReading an AsyncIO futureAsyncIO for networkingUsing executors to wrap blocking codeStreamsExecutors
Case study
2. Course Module 2: Data Analysis
1. Introducing Data Analysis and Libraries
Data analysis and processing
An overview of the libraries in data analysis
Python libraries in data analysis
NumPypandasMatplotlibPyMongoThe scikit-learn library
2. NumPy Arrays and Vectorized Computation
NumPy arraysData typesArray creationIndexing and slicingFancy indexingNumerical operations on arrays
Array functions
Data processing using arrays
Loading and saving dataSaving an arrayLoading an array
Linear algebra with NumPy
NumPy random numbers
3. Data Analysis with pandas
An overview of the pandas package
The pandas data structure
SeriesThe DataFrame
The essential basic functionality
Reindexing and altering labelsHead and tailBinary operationsFunctional statisticsFunction applicationSorting
Indexing and selecting data
Computational tools
Working with missing data
Advanced uses of pandas for data analysis
Hierarchical indexingThe Panel data
4. Data Visualization
The matplotlib API primerLine propertiesFigures and subplots
Exploring plot types
Scatter plotsBar plotsContour plotsHistogram plots
Legends and annotations
Plotting functions with pandas
Additional Python data visualization tools
BokehMayaVi
5. Time Series
Time series primer
Working with date and time objects
Resampling time series
Downsampling time series data
Upsampling time series data
Timedeltas
Time series plotting
6. Interacting with Databases
Interacting with data in text formatReading data from text formatWriting data to text format
Interacting with data in binary format
HDF5
Interacting with data in MongoDB
Interacting with data in Redis
The simple valueListSetOrdered set
7. Data Analysis Application Examples
Data mungingCleaning dataFilteringMerging dataReshaping data
Data aggregation
Grouping data
3. Course Module 3: Data Mining
1. Getting Started with Data Mining
Introducing data mining
A simple affinity analysis example
What is affinity analysis?Product recommendationsLoading the dataset with NumPyImplementing a simple ranking of rulesRanking to find the best rules
A simple classification example
What is classification?
Loading and preparing the datasetImplementing the OneR algorithmTesting the algorithm
2. Classifying with scikit-learn Estimators
scikit-learn estimatorsNearest neighborsDistance metricsLoading the datasetMoving towards a standard workflowRunning the algorithmSetting parameters
Preprocessing using pipelines
An exampleStandard preprocessingPutting it all together
Pipelines
3. Predicting Sports Winners with Decision Trees
Loading the datasetCollecting the dataUsing pandas to load the datasetCleaning up the datasetExtracting new features
Decision trees
Parameters in decision treesUsing decision trees
Sports outcome prediction
Putting it all together
Random forests
How do ensembles work?Parameters in Random forestsApplying Random forestsEngineering new features
4. Recommending Movies Using Affinity Analysis
Affinity analysisAlgorithms for affinity analysisChoosing parameters
The movie recommendation problem
Obtaining the datasetLoading with pandasSparse data formats
The Apriori implementation
The Apriori algorithmImplementation
Extracting association rules
Evaluation
5. Extracting Features with Transformers
Feature extractionRepresenting reality in modelsCommon feature patternsCreating good features
Feature selection
Selecting the best individual features
Feature creation
Creating your own transformer
The transformer APIImplementation detailsUnit testingPutting it all together
6. Social Media Insight Using Naive Bayes
DisambiguationDownloading data from a social networkLoading and classifying the datasetCreating a replicable dataset from Twitter
Text transformers
Bag-of-wordsN-gramsOther features
Naive Bayes
Bayes' theoremNaive Bayes algorithmHow it works
Application
Extracting word countsConverting dictionaries to a matrixTraining the Naive Bayes classifierPutting it all togetherEvaluation using the F1-scoreGetting useful features from models
7. Discovering Accounts to Follow Using Graph Mining
Loading the datasetClassifying with an existing modelGetting follower information from TwitterBuilding the networkCreating a graphCreating a similarity graph
Finding subgraphs
Connected componentsOptimizing criteria
8. Beating CAPTCHAs with Neural Networks
Artificial neural networksAn introduction to neural networks
Creating the dataset
Drawing basic CAPTCHAsSplitting the image into individual lettersCreating a training datasetAdjusting our training dataset to our methodology
Training and classifying
Back propagationPredicting words
Improving accuracy using a dictionary
Ranking mechanisms for wordsPutting it all together
9. Authorship Attribution
Attributing documents to authorsApplications and use casesAttributing authorshipGetting the data
Function words
Counting function wordsClassifying with function words
Support vector machines
Classifying with SVMsKernels
Character n-grams
Extracting character n-grams
Using the Enron dataset
Accessing the Enron datasetCreating a dataset loaderPutting it all togetherEvaluation
10. Clustering News Articles
Obtaining news articlesUsing a Web API to get dataReddit as a data sourceGetting the data
Extracting text from arbitrary websites
Finding the stories in arbitrary websitesPutting it all together
Grouping news articles
The k-means algorithmEvaluating the resultsExtracting topic information from clustersUsing clustering algorithms as transformers
Clustering ensembles
Evidence accumulationHow it worksImplementation
Online learning
An introduction to online learningImplementation
11. Classifying Objects in Images Using Deep Learning
Object classification
Application scenario and goals
Use cases
Deep neural networks
IntuitionImplementationAn introduction to TheanoAn introduction to LasagneImplementing neural networks with nolearn
GPU optimization
When to use GPUs for computationRunning our code on a GPU
Setting up the environment
Application
Getting the dataCreating the neural networkPutting it all together
12. Working with Big Data
Big data
Application scenario and goals
MapReduce
IntuitionA word count exampleHadoop MapReduce
Application
Getting the dataNaive Bayes predictionThe mrjob packageExtracting the blog postsTraining Naive BayesPutting it all togetherTraining on Amazon's EMR infrastructure
13. Next Steps…
Chapter 1 – Getting Started with Data MiningScikit-learn tutorialsExtending the IPython Notebook
Chapter 2 – Classifying with scikit-learn Estimators
More complex pipelinesComparing classifiers
Chapter 3: Predicting Sports Winners with Decision Trees
More on pandas
Chapter 4 – Recommending Movies Using Affinity Analysis
The Eclat algorithm
Chapter 5 – Extracting Features with Transformers
Vowpal Wabbit
Chapter 6 – Social Media Insight Using Naive Bayes
Natural language processing and part-of-speech tagging
Chapter 7 – Discovering Accounts to Follow Using Graph Mining
More complex algorithms
Chapter 8 – Beating CAPTCHAs with Neural Networks
Deeper networksReinforcement learning
Chapter 9 – Authorship Attribution
Local n-grams
Chapter 10 – Clustering News Articles
Real-time clusterings
Chapter 11 – Classifying Objects in Images Using Deep Learning
Keras and Pylearn2Mahotas
Chapter 12 – Working with Big Data
Courses on HadoopPydoopRecommendation engine
More resources
4. Course Module 4: Machine Learning
1. Giving Computers the Ability to Learn from Data
How to transform data into knowledge
The three different types of machine learning
Making predictions about the future with supervised learningClassification for predicting class labelsRegression for predicting continuous outcomesSolving interactive problems with reinforcement learningDiscovering hidden structures with unsupervised learningFinding subgroups with clusteringDimensionality reduction for data compression
An introduction to the basic terminology and notations
A roadmap for building machine learning systems
Preprocessing – getting data into shapeTraining and selecting a predictive modelEvaluating models and predicting unseen data instances
Using Python for machine learning
2. Training Machine Learning Algorithms for Classification
Artificial neurons – a brief glimpse into the early history of machine learning
Implementing a perceptron learning algorithm in Python
Training a perceptron model on the Iris dataset
Adaptive linear neurons and the convergence of learning
Minimizing cost functions with gradient descentImplementing an Adaptive Linear Neuron in PythonLarge scale machine learning and stochastic gradient descent
3. A Tour of Machine Learning Classifiers Using scikit-learn
Choosing a classification algorithm
First steps with scikit-learn
Training a perceptron via scikit-learn
Modeling class probabilities via logistic regression
Logistic regression intuition and conditional probabilitiesLearning the weights of the logistic cost functionTraining a logistic regression model with scikit-learnTackling overfitting via regularization
Maximum margin classification with support vector machines
Maximum margin intuitionDealing with the nonlinearly separable case using slack variablesAlternative implementations in scikit-learn
Solving nonlinear problems using a kernel SVM
Using the kernel trick to find separating hyperplanes in higher dimensional space
Decision tree learning
Maximizing information gain – getting the most bang for the buckBuilding a decision treeCombining weak to strong learners via random forests
K-nearest neighbors – a lazy learning algorithm
4. Building Good Training Sets – Data Preprocessing
Dealing with missing dataEliminating samples or features with missing valuesImputing missing valuesUnderstanding the scikit-learn estimator API
Handling categorical data
Mapping ordinal featuresEncoding class labelsPerforming one-hot encoding on nominal features
Partitioning a dataset in training and test sets
Bringing features onto the same scale
Selecting meaningful features
Sparse solutions with L1 regularizationSequential feature selection algorithms
Assessing feature importance with random forests
5. Compressing Data via Dimensionality Reduction
Unsupervised dimensionality reduction via principal component analysisTotal and explained varianceFeature transformationPrincipal component analysis in scikit-learn
Supervised data compression via linear discriminant analysis
Computing the scatter matricesSelecting linear discriminants for the new feature subspaceProjecting samples onto the new feature spaceLDA via scikit-learn
Using kernel principal component analysis for nonlinear mappings
Kernel functions and the kernel trickImplementing a kernel principal component analysis in PythonExample 1 – separating half-moon shapesExample 2 – separating concentric circlesProjecting new data pointsKernel principal component analysis in scikit-learn
6. Learning Best Practices for Model Evaluation and Hyperparameter Tuning
Streamlining workflows with pipelinesLoading the Breast Cancer Wisconsin datasetCombining transformers and estimators in a pipeline
Using k-fold cross-validation to assess model performance
The holdout methodK-fold cross-validation
Debugging algorithms with learning and validation curves
Diagnosing bias and variance problems with learning curvesAddressing overfitting and underfitting with validation curves
Fine-tuning machine learning models via grid search
Tuning hyperparameters via grid searchAlgorithm selection with nested cross-validation
Looking at different performance evaluation metrics
Reading a confusion matrixOptimizing the precision and recall of a classification modelPlotting a receiver operating characteristicThe scoring metrics for multiclass classification
7. Combining Different Models for Ensemble Learning
Learning with ensembles
Implementing a simple majority vote classifier
Combining different algorithms for classification with majority vote
Evaluating and tuning the ensemble classifier
Bagging – building an ensemble of classifiers from bootstrap samples
Leveraging weak learners via adaptive boosting
8. Predicting Continuous Target Variables with Regression Analysis
Introducing a simple linear regression model
Exploring the Housing Dataset
Visualizing the important characteristics of a dataset
Implementing an ordinary least squares linear regression model
Solving regression for regression parameters with gradient descentEstimating the coefficient of a regression model via scikit-learn
Fitting a robust regression model using RANSAC
Evaluating the performance of linear regression models
Using regularized methods for regression
Turning a linear regression model into a curve – polynomial regression
Modeling nonlinear relationships in the Housing DatasetDealing with nonlinear relationships using random forestsDecision tree regressionRandom forest regression
A. Reflect and Test Yourself! Answers
Module 2: Data AnalysisChapter 1: Introducing Data Analysis and LibrariesChapter 2: Object-oriented DesignChapter 3: Data Analysis with pandasChapter 4: Data VisualizationChapter 5: Time SeriesChapter 6: Interacting with DatabasesChapter 7: Data Analysis Application Examples
Module 3: Data Mining
Chapter 1: Getting Started with Data MiningChapter 2: Classifying with scikit-learn EstimatorsChapter 3: Predicting Sports Winners with Decision TreesChapter 4: Recommending Movies Using Affinity AnalysisChapter 5: Extracting Features with TransformersChapter 6: Social Media Insight Using Naive BayesChapter 7: Discovering Accounts to Follow Using Graph MiningChapter 8: Beating CAPTCHAs with Neural NetworksChapter 9: Authorship AttributionChapter 10: Clustering News ArticlesChapter 11: Classifying Objects in Images Using Deep LearningChapter 12: Working with Big Data
Module 4: Machine Learning
Chapter 1: Giving Computers the Ability to Learn from DataChapter 2: Training Machine LearningChapter 3: A Tour of Machine Learning Classifiers Using scikit-learnChapter 4: Building Good Training Sets – Data PreprocessingChapter 5: Compressing Data via Dimensionality ReductionChapter 6: Learning Best Practices for Model Evaluation and Hyperparameter TuningChapter 7: Combining Different Models for Ensemble LearningChapter 8: Predicting Continuous Target Variables with Regression Analysis
B. Bibliography
Index

Content preview from Python: Real-World Data Science

Chapter 12. Working with Big Data

The amount of data is increasing at exponential rates. Today's systems are generating and recording information on customer behavior, distributed systems, network analysis, sensors and many, many more sources. While the current big trend of mobile data is pushing the current growth, the next big thing—the Internet of Things (IoT)—is going to further increase the rate of growth.

What this means for data mining is a new way of thinking. The complex algorithms with high run times need to be improved or discarded, while simpler algorithms that can deal with more samples are becoming more popular to use. As an example, while support vector machines are great classifiers, some variants are difficult to use on very large ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781786465160

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Python: Real-World Data Science

by Dusty Phillips, Fabrizio Romano, Phuong Vo.T.H, Martin Czygan, Robert Layton, Sebastian Raschka

Chapter 12. Working with Big Data

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

More than 5,000 organizations count on O’Reilly

Julian F.

Addison B.

Amir M.

Mark W.

You might also like

Practical Data Science with Python

Python for Data Science

Python Data Science Essentials - Third Edition

Data Science Projects with Python - Second Edition

Publisher Resources