book

Data Science with Python and Dask

by Jesse Daniel

July 2019

Intermediate to advanced

296 pages

9h 1m

English

Manning Publications

Read now

Unlock full access

Who should read this bookHow this book is organized: A roadmapAbout the code liveBook discussion forum

1.1 Why Dask?1.2 Cooking with DAGs1.3 Scaling out, concurrency, and recovery1.3.1 Scaling up vs. scaling out1.3.2 Concurrency and resource management1.3.3 Recovering from failures1.4 Introducing a companion datasetSummary
2.1 Hello Dask: A first look at the DataFrame API2.1.1 Examining the metadata of Dask objects2.1.2 Running computations with the compute method2.1.3 Making complex computations more efficient with persist2.2 Visualizing DAGs2.2.1 Visualizing a simple DAG using Dask Delayed objects2.2.2 Visualizing more complex DAGs with loops and collections2.2.3 Reducing DAG complexity with persist2.3 Task scheduling2.3.1 Lazy computations2.3.2 Data localitySummary
3.1 Why use DataFrames?3.2 Dask and Pandas3.2.1 Managing DataFrame partitioning3.2.2 What is the shuffle?3.3 Limitations of Dask DataFramesSummary
4.1 Reading data from text files4.1.1 Using Dask datatypes4.1.2 Creating schemas for Dask DataFrames4.2 Reading data from relational databases4.3 Reading data from HDFS and S34.4 Reading data in Parquet formatSummary
5.1 Working with indexes and axes5.1.1 Selecting columns from a DataFrame5.1.2 Dropping columns from a DataFrame5.1.3 Renaming columns in a DataFrame5.1.4 Selecting rows from a DataFrame5.2 Dealing with missing values5.2.1 Counting missing values in a DataFrame5.2.2 Dropping columns with missing values5.2.3 Imputing missing values5.2.4 Dropping rows with missing data5.2.5 Imputing multiple columns with missing values5.3 Recoding data5.4 Elementwise operations5.5 Filtering and reindexing DataFrames5.6 Joining and concatenating DataFrames5.6.1 Joining two DataFrames5.6.2 Unioning two DataFrames5.7 Writing data to text files and Parquet files5.7.1 Writing to delimited text files5.7.2 Writing to Parquet filesSummary
6.1 Descriptive statistics6.1.1 What are descriptive statistics?6.1.2 Calculating descriptive statistics with Dask6.1.3 Using the describe method for descriptive statistics6.2 Built-In aggregate functions6.2.1 What is correlation?6.2.2 Calculating correlations with Dask DataFrames6.3 Custom aggregate functions6.3.1 Testing categorical variables with the t-test6.3.2 Using custom aggregates to implement the Brown-Forsythe test6.4 Rolling (window) functions6.4.1 Preparing data for a rolling function6.4.2 Using the rolling method to apply a window functionSummary
7.1 The prepare-reduce-collect-plot pattern7.2 Visualizing continuous relationships with scatterplot and regplot7.2.1 Creating a scatterplot with Dask and Seaborn7.2.2 Adding a linear regression line to the scatterplot7.2.3 Adding a nonlinear regression line to a scatterplot7.3 Visualizing categorical relationships with violinplot7.3.1 Creating a violinplot with Dask and Seaborn7.3.2 Randomly sampling data from a Dask DataFrame7.4 Visualizing two categorical relationships with heatmapSummary
8.1 What is Datashader and how does it work?8.1.1 The five stages of the Datashader rendering pipeline8.1.2 Creating a Datashader Visualization8.2 Plotting location data as an interactive heatmap8.2.1 Preparing geographic data for map tiling8.2.2 Creating the interactive heatmapSummary
9.1 Reading and parsing unstructured data with Bags9.1.1 Selecting and viewing data from a Bag9.1.2 Common parsing issues and how to overcome them9.1.3 Working with delimiters9.2 Transforming, filtering, and folding elements9.2.1 Transforming elements with the map method9.2.2 Filtering Bags with the filter method9.2.3 Calculating descriptive statistics on Bags9.2.4 Creating aggregate functions using the foldby method9.3 Building Arrays and DataFrames from Bags9.4 Using Bags for parallel text analysis with NLTK9.4.1 The basics of bigram analysis9.4.2 Extracting tokens and filtering stopwords9.4.3 Analyzing the bigramsSummary
10.1 Building linear models with Dask-ML10.1.1 Preparing the data with binary vectorization10.1.2 Building a logistic regression model with Dask-ML10.2 Evaluating and tuning Dask-ML models10.2.1 Evaluating Dask-ML models with the score method10.2.2 Building a naïve Bayes classifier with Dask-ML10.2.3 Automatically tuning hyperparameters10.3 Persisting Dask-ML modelsSummary
11.1 Building a Dask cluster on Amazon AWS with Docker 11.1.1 Getting started11.1.2 Creating a security key11.1.3 Creating the ECS cluster11.1.4 Configuring the cluster’s networking11.1.5 Creating a shared data drive in Elastic File System11.1.6 Allocating space for Docker images in Elastic Container Repository11.1.7 Building and deploying images for scheduler, worker, and notebook11.1.8 Connecting to the cluster11.2 Running and monitoring Dask jobs on a cluster11.3 Cleaning up the Dask cluster on AWSSummary
Installing additional packages with AnacondaInstalling packages without AnacondaStarting a Jupyter Notebook serverConfiguring NLTK
List of TablesList of Listings

Content preview from Data Science with Python and Dask

5 Cleaning and transforming DataFrames

This chapter covers

Selecting and filtering data
Creating and dropping columns
Finding and fixing columns with missing values
Indexing and sorting DataFrames
Combining DataFrames using join and union operations
Writing DataFrames to delimited text files and Parquet

In the previous chapter, we created a schema for the NYC Parking Ticket dataset and successfully loaded the data into Dask. Now we’re ready to get the data cleaned up so we can begin analyzing and visualizing it! As a friendly reminder, figure 5.1 shows what we’ve done so far and where we’re going next within our data science workflow.

Figure 5.1 The Data Science with Python and Dask workflow

Data cleaning is an important part of any ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Start your free trial

Practical Data Science with Python

Nathan George

Python: End-to-end Data Analysis

Phuong Vothihong, Martin Czygan, Ivan Idris, Magnus Vilhelm Persson, Luiz Felipe Martins

Python for Data Science

Yuli Vasiliev

Machine Learning with Python Cookbook

Chris Albon

Publisher Resources

ISBN: 9781617295607Other Supplemental Content Publisher Support Publisher Website Supplemental Content Purchase Link