book

Hands-On Data Preprocessing in Python

Name: Hands-On Data Preprocessing in Python
Author: Roy Jafari
ISBN: 9781801072137

by Roy Jafari

January 2022

Beginner to intermediate

602 pages

13h 51m

English

Packt Publishing

Read now

Unlock full access

Hands-On Data Preprocessing in Python
ContributorsAbout the authorAbout the reviewers
Preface
Who this book is forWhat this book coversTo get the most out of this bookDownload the example code filesDownload the color imagesConventions usedGet in touchShare Your Thoughts
Part 1:Technical Needs
Chapter 1: Review of the Core Modules of NumPy and Pandas
Technical requirementsOverview of the Jupyter NotebookAre we analyzing data via computer programming?Overview of the basic functions of NumPyThe np.arange() functionThe np.zeros() and np.ones() functionsThe np.linspace() functionOverview of PandasPandas data accessBoolean masking for filtering a DataFramePandas functions for exploring a DataFramePandas applying a functionThe Pandas groupby functionPandas multi-level indexingPandas pivot and melt functionsSummaryExercises
Chapter 2: Review of Another Core Module – Matplotlib
Technical requirementsDrawing the main plots in MatplotlibSummarizing numerical attributes using histograms or boxplotsObserving trends in the data using a line plotRelating two numerical attributes using a scatterplotModifying the visualsAdding a title to visuals and labels to the axisAdding legendsModifying ticksModifying markersSubplotsResizing visuals and saving themResizingSavingExample of Matplotilb assisting data preprocessingSummaryExercises
Chapter 3: Data – What Is It Really?
Technical requirementsWhat is data?Why this definition?DIKW pyramidData preprocessing for data analytics versus data preprocessing for machine learningThe most universal data structure – a tableData objectsData attributesTypes of data valuesAnalytics standpointProgramming standpointInformation versus patternUnderstanding everyday use of the word "information"Statistical use of the word "information"Statistical meaning of the word "pattern"SummaryExercisesReferences
Chapter 4: Databases
Technical requirementsWhat is a database?Understanding the difference between a database and a datasetTypes of databasesThe differentiating elements of databasesRelational databases (SQL databases)Unstructured databases (NoSQL databases)A practical example that requires a combination of both structured and unstructured databasesDistributed databasesBlockchainConnecting to, and pulling data from, databasesDirect connectionWeb page connectionAPI connectionRequest connectionPublicly sharedSummaryExercises
Part 2: Analytic Goals
Chapter 5: Data Visualization
Technical requirementsSummarizing a populationExample of summarizing numerical attributesExample of summarizing categorical attributesComparing populationsExample of comparing populations using boxplotsExample of comparing populations using histogramsExample of comparing populations using bar chartsInvestigating the relationship between two attributesVisualizing the relationship between two numerical attributesVisualizing the relationship between two categorical attributesVisualizing the relationship between a numerical attribute and a categorical attributeAdding visual dimensionsExample of a five-dimensional scatter plotShowing and comparing trendsExample of visualizing and comparing trendsSummaryExercise
Chapter 6: Prediction
Technical requirementsPredictive modelsForecastingRegression analysisLinear regressionExample of applying linear regression to perform regression analysisMLPHow does MLP work?Example of applying MLP to perform regression analysisSummaryExercises

Chapter 7: Classification
Technical requirementsClassification modelsExample of designing a classification modelClassification algorithmsKNNExample of using KNN for classificationDecision TreesExample of using Decision Trees for classificationSummaryExercises
Chapter 8: Clustering Analysis
Technical requirementsClustering modelClustering example using a two-dimensional datasetClustering example using a three-dimensional datasetK-Means algorithmUsing K-Means to cluster a two-dimensional datasetUsing K-Means to cluster a dataset with more than two dimensionsCentroid analysisSummaryExercises
Part 3: The Preprocessing
Chapter 9: Data Cleaning Level I – Cleaning Up the Table
Technical requirementsThe levels, tools, and purposes of data cleaning – a roadmap to chapters 9, 10, and 11Purpose of data analyticsTools for data analyticsLevels of data cleaningMapping the purposes and tools of analytics to the levels of data cleaningData cleaning level I – cleaning up the tableExample 1 – unwise data collectionExample 2 – reindexing (multi-level indexing)Example 3 – intuitive but long column titlesSummaryExercises
Chapter 10: Data Cleaning Level II – Unpacking, Restructuring, and Reformulating the Table
Technical requirementsExample 1 – unpacking columns and reformulating the tableUnpacking FileNameUnpacking ContentReformulating a new table for visualizationThe last step – drawing the visualizationExample 2 – restructuring the tableExample 3 – level I and II data cleaningLevel I cleaningLevel II cleaningDoing the analytics – using linear regression to create a predictive modelSummaryExercises
Chapter 11: Data Cleaning Level III – Missing Values, Outliers, and Errors
Technical requirementsMissing valuesDetecting missing valuesExample of detecting missing valuesCauses of missing valuesTypes of missing valuesDiagnosis of missing valuesDealing with missing valuesOutliersDetecting outliersDealing with outliersErrors Types of errorsDealing with errorsDetecting systematic errorsSummaryExercises
Chapter 12: Data Fusion and Data Integration
Technical requirementsWhat are data fusion and data integration?Data fusion versus data integrationDirections of data integrationFrequent challenges regarding data fusion and integrationChallenge 1 – entity identificationChallenge 2 – unwise data collectionChallenge 3 – index mismatched formattingChallenge 4 – aggregation mismatchChallenge 5 – duplicate data objectsChallenge 6 – data redundancyExample 1 (challenges 3 and 4)Example 2 (challenges 2 and 3)Example 3 (challenges 1, 3, 5, and 6)Checking for duplicate data objectsDesigning the structure for the result of data integrationFilling songIntegrate_df from billboard_dfFilling songIntegrate_df from songAttribute_dfFilling songIntegrate_df from artist_dfChecking for data redundancyThe analysisExample summarySummary Exercise
Chapter 13: Data Reduction
Technical requirementsThe distinction between data reduction and data redundancyThe objectives of data reductionTypes of data reductionPerforming numerosity data reductionRandom samplingStratified samplingRandom over/undersamplingPerforming dimensionality data reductionLinear regression as a dimension reduction methodUsing a decision tree as a dimension reduction methodUsing random forest as a dimension reduction methodBrute-force computational dimension reductionPCAFunctional data analysisSummaryExercises
Chapter 14: Data Transformation and Massaging
Technical requirementsThe whys of data transformation and massagingData transformation versus data massagingNormalization and standardizationBinary coding, ranking transformation, and discretizationExample one – binary coding of nominal attributeExample two – binary coding or ranking transformation of ordinal attributesExample three – discretization of numerical attributesUnderstanding the types of discretizationDiscretization – the number of cut-off pointsA summary – from numbers to categories and backAttribute constructionExample – construct one transformed attribute from two attributesFeature extractionExample – extract three attributes from one attributeExample – Morphological feature extractionFeature extraction examples from the previous chaptersLog transformationImplementation – doing it yourselfImplementation – the working module doing it for youSmoothing, aggregation, and binningSmoothingAggregationBinningSummaryExercise
Part 4: Case Studies
Chapter 15: Case Study 1 – Mental Health in Tech
Technical requirementsIntroducing the case studyThe audience of the results of analyticsIntroduction to the source of the dataIntegrating the data sourcesCleaning the dataDetecting and dealing with outliers and errorsDetecting and dealing with missing valuesAnalyzing the dataAnalysis question one – is there a significant difference between the mental health of employees across the attribute of gender?Analysis question two – is there a significant difference between the mental health of employees across the Age attribute?Analysis question three – do more supportive companies have mentally healthier employees?Analysis question four – does the attitude of individuals toward mental health influence their mental health and their seeking of treatments?Summary
Chapter 16: Case Study 2 – Predicting COVID-19 Hospitalizations
Technical requirementsIntroducing the case studyIntroducing the source of the dataPreprocessing the dataDesigning the dataset to support the predictionFilling up the placeholder datasetSupervised dimension reductionAnalyzing the dataSummary
Chapter 17: Case Study 3: United States Counties Clustering Analysis
Technical requirementsIntroducing the case studyIntroduction to the source of the dataPreprocessing the dataTransforming election_df to partisan_dfCleaning edu_df, employ_df, pop_df, and pov_df Data integrationData cleaning level III – missing values, errors, and outliersChecking for data redundancyAnalyzing the dataUsing PCA to visualize the datasetK-Means clustering analysisSummary
Chapter 18: Summary, Practice Case Studies, and Conclusions
A summary of the bookPart 1 – Technical requirementsPart 2 – Analytics goalsPart 3 – The preprocessingPart 4 – Case studiesPractice case studiesGoogle Covid-19 mobility datasetPolice killings in the USUS accidentsSan Francisco crimeData analytics job marketFIFA 2018 player of the matchHot hands in basketballWildfires in CaliforniaSilicon Valley diversity profileRecognizing fake job postingHunting more practice case studiesConclusions
Why subscribe?
Other Books You May EnjoyPackt is searching for authors like youShare Your Thoughts

Content preview from Hands-On Data Preprocessing in Python

Chapter 11: Data Cleaning Level III – Missing Values, Outliers, and Errors

In level I, we cleaned up the table without paying attention to the data structure or the recorded values. In level II, our attention was to have a data structure that would support our analytic goal, but we still didn't pay much attention to the correctness or appropriateness of the recorded values. That is the objective of data cleaning level III. In data cleaning level III, we will focus on the recorded values and will take measures to make sure that three matters regarding the values recorded in the data are addressed. First, we will make sure missing values in the data have been detected, that we know why this has happened, and that appropriate measures have been ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781801072137

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Hands-On Data Preprocessing in Python

by Roy Jafari

Chapter 11: Data Cleaning Level III – Missing Values, Outliers, and Errors

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.