O'Reilly logo

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Practical Data Analysis

Book Description

For small businesses, analyzing the information contained in their data using open source technology could be game-changing. All you need is some basic programming and mathematical skills to do just that.

  • Explore how to analyze your data in various innovative ways and turn them into insight
  • Learn to use the D3.js visualization tool for exploratory data analysis
  • Understand how to work with graphs and social data analysis
  • Discover how to perform advanced query techniques and run MapReduce on MongoDB

In Detail

Plenty of small businesses face big amounts of data but lack the internal skills to support quantitative analysis. Understanding how to harness the power of data analysis using the latest open source technology can lead them to providing better customer service, the visualization of customer needs, or even the ability to obtain fresh insights about the performance of previous products. Practical Data Analysis is a book ideal for home and small business users who want to slice and dice the data they have on hand with minimum hassle.

Practical Data Analysis is a hands-on guide to understanding the nature of your data and turn it into insight. It will introduce you to the use of machine learning techniques, social networks analytics, and econometrics to help your clients get insights about the pool of data they have at hand. Performing data preparation and processing over several kinds of data such as text, images, graphs, documents, and time series will also be covered.

Practical Data Analysis presents a detailed exploration of the current work in data analysis through self-contained projects. First you will explore the basics of data preparation and transformation through OpenRefine. Then you will get started with exploratory data analysis using the D3js visualization framework. You will also be introduced to some of the machine learning techniques such as, classification, regression, and clusterization through practical projects such as spam classification, predicting gold prices, and finding clusters in your Facebook friends’ network. You will learn how to solve problems in text classification, simulation, time series forecast, social media, and MapReduce through detailed projects. Finally you will work with large amounts of Twitter data using MapReduce to perform a sentiment analysis implemented in Python and MongoDB.

Practical Data Analysis contains a combination of carefully selected algorithms and data scrubbing that enables you to turn your data into insight.

Table of Contents

  1. Practical Data Analysis
    1. Table of Contents
    2. Practical Data Analysis
    3. Credits
    4. Foreword
    5. About the Author
    6. Acknowledgments
    7. About the Reviewers
    8. www.PacktPub.com
      1. Support files, eBooks, discount offers and more
        1. Why Subscribe?
        2. Free Access for Packt account holders
    9. Preface
      1. What this book covers
      2. What you need for this book
      3. Who this book is for
      4. Conventions
      5. Reader feedback
      6. Customer support
        1. Downloading the example code
        2. Errata
        3. Piracy
        4. Questions
    10. 1. Getting Started
      1. Computer science
      2. Artificial intelligence (AI)
      3. Machine Learning (ML)
      4. Statistics
      5. Mathematics
      6. Knowledge domain
      7. Data, information, and knowledge
      8. The nature of data
      9. The data analysis process
        1. The problem
        2. Data preparation
        3. Data exploration
        4. Predictive modeling
        5. Visualization of results
      10. Quantitative versus qualitative data analysis
      11. Importance of data visualization
      12. What about big data?
        1. Sensors and cameras
        2. Social networks analysis
        3. Tools and toys for this book
        4. Why Python?
        5. Why mlpy?
        6. Why D3.js?
        7. Why MongoDB?
      13. Summary
    11. 2. Working with Data
      1. Datasource
        1. Open data
        2. Text files
        3. Excel files
        4. SQL databases
        5. NoSQL databases
        6. Multimedia
        7. Web scraping
      2. Data scrubbing
        1. Statistical methods
        2. Text parsing
        3. Data transformation
      3. Data formats
        1. CSV
          1. Parsing a CSV file with the csv module
          2. Parsing a CSV file using NumPy
        2. JSON
          1. Parsing a JSON file using json module
        3. XML
          1. Parsing an XML file in Python using xml module
        4. YAML
      4. Getting started with OpenRefine
        1. Text facet
        2. Clustering
        3. Text filters
        4. Numeric facets
        5. Transforming data
        6. Exporting data
        7. Operation history
      5. Summary
    12. 3. Data Visualization
      1. Data-Driven Documents (D3)
        1. HTML
        2. DOM
        3. CSS
        4. JavaScript
        5. SVG
      2. Getting started with D3.js
        1. Bar chart
        2. Pie chart
        3. Scatter plot
        4. Single line chart
        5. Multi-line chart
      3. Interaction and animation
      4. Summary
    13. 4. Text Classification
      1. Learning and classification
      2. Bayesian classification
        1. Naïve Bayes algorithm
      3. E-mail subject line tester
      4. The algorithm
      5. Classifier accuracy
      6. Summary
    14. 5. Similarity-based Image Retrieval
      1. Image similarity search
      2. Dynamic time warping (DTW)
      3. Processing the image dataset
      4. Implementing DTW
      5. Analyzing the results
      6. Summary
    15. 6. Simulation of Stock Prices
      1. Financial time series
      2. Random walk simulation
      3. Monte Carlo methods
      4. Generating random numbers
      5. Implementation in D3.js
      6. Summary
    16. 7. Predicting Gold Prices
      1. Working with the time series data
        1. Components of a time series
      2. Smoothing the time series
      3. The data – historical gold prices
      4. Nonlinear regression
        1. Kernel ridge regression
        2. Smoothing the gold prices time series
        3. Predicting in the smoothed time series
        4. Contrasting the predicted value
      5. Summary
    17. 8. Working with Support Vector Machines
      1. Understanding the multivariate dataset
      2. Dimensionality reduction
        1. Linear Discriminant Analysis
        2. Principal Component Analysis
      3. Getting started with support vector machine
        1. Kernel functions
        2. Double spiral problem
        3. SVM implemented on mlpy
      4. Summary
    18. 9. Modeling Infectious Disease with Cellular Automata
      1. Introduction to epidemiology
        1. The epidemiology triangle
      2. The epidemic models
        1. The SIR model
        2. Solving ordinary differential equation for the SIR model with SciPy
        3. The SIRS model
      3. Modeling with cellular automata
        1. Cell, state, grid, and neighborhood
        2. Global stochastic contact model
      4. Simulation of the SIRS model in CA with D3.js
      5. Summary
    19. 10. Working with Social Graphs
      1. Structure of a graph
        1. Undirected graph
        2. Directed graph
      2. Social Networks Analysis
      3. Acquiring my Facebook graph
        1. Using Netvizz
      4. Representing graphs with Gephi
      5. Statistical analysis
        1. Male to female ratio
      6. Degree distribution
        1. Histogram of a graph
        2. Centrality
      7. Transforming GDF to JSON
      8. Graph visualization with D3.js
      9. Summary
    20. 11. Sentiment Analysis of Twitter Data
      1. The anatomy of Twitter data
        1. Tweet
        3. Trending topics
      2. Using OAuth to access Twitter API
      3. Getting started with Twython
        1. Simple search
        2. Working with timelines
        3. Working with followers
        4. Working with places and trends
      4. Sentiment classification
        1. Affective Norms for English Words
        2. Text corpus
      5. Getting started with Natural Language Toolkit (NLTK)
        1. Bag of words
        2. Naive Bayes
        3. Sentiment analysis of tweets
      6. Summary
    21. 12. Data Processing and Aggregation with MongoDB
      1. Getting started with MongoDB
        1. Database
        2. Collection
        3. Document
        4. Mongo shell
        5. Insert/Update/Delete
        6. Queries
      2. Data preparation
        1. Data transformation with OpenRefine
        2. Inserting documents with PyMongo
      3. Group
      4. The aggregation framework
        1. Pipelines
        2. Expressions
      5. Summary
    22. 13. Working with MapReduce
      1. MapReduce overview
      2. Programming model
      3. Using MapReduce with MongoDB
        1. The map function
        2. The reduce function
        3. Using mongo shell
        4. Using UMongo
        5. Using PyMongo
      4. Filtering the input collection
      5. Grouping and aggregation
      6. Word cloud visualization of the most common positive words in tweets
      7. Summary
    23. 14. Online Data Analysis with IPython and Wakari
      1. Getting started with Wakari
        1. Creating an account in Wakari
      2. Getting started with IPython Notebook
        1. Data visualization
      3. Introduction to image processing with PIL
        1. Opening an image
        2. Image histogram
        3. Filtering
        4. Operations
        5. Transformations
      4. Getting started with Pandas
        1. Working with time series
        2. Working with multivariate dataset with DataFrame
        3. Grouping, aggregation, and correlation
      5. Multiprocessing with IPython
        1. Pool
      6. Sharing your Notebook
        1. The data
      7. Summary
    24. A. Setting Up the Infrastructure
      1. Installing and running Python 3
        1. Installing and running Python 3.2 on Ubuntu
        2. Installing and running IDLE on Ubuntu
        3. Installing and running Python 3.2 on Windows
        4. Installing and running IDLE on Windows
      2. Installing and running NumPy
        1. Installing and running NumPy on Ubuntu
        2. Installing and running NumPy on Windows
      3. Installing and running SciPy
        1. Installing and running SciPy on Ubuntu
        2. Installing and running SciPy on Windows
      4. Installing and running mlpy
        1. Installing and running mlpy on Ubuntu
        2. Installing and running mlpy on Windows
      5. Installing and running OpenRefine
        1. Installing and running OpenRefine on Linux
        2. Installing and running OpenRefine on Windows
      6. Installing and running MongoDB
        1. Installing and running MongoDB on Ubuntu
        2. Installing and running MongoDB on Windows
        3. Connecting Python with MongoDB
      7. Installing and running UMongo
        1. Installing and running Umongo on Ubuntu
        2. Installing and running Umongo on Windows
      8. Installing and running Gephi
        1. Installing and running Gephi on Linux
        2. Installing and running Gephi on Windows
    25. Index