book

Hands-On Data Preprocessing in Python

by Roy Jafari

January 2022

Beginner to intermediate

602 pages

13h 51m

English

Packt Publishing

Read now

Unlock full access

ContributorsAbout the authorAbout the reviewers
Who this book is forWhat this book coversTo get the most out of this bookDownload the example code filesDownload the color imagesConventions usedGet in touchShare Your Thoughts
Technical requirementsOverview of the Jupyter NotebookAre we analyzing data via computer programming?Overview of the basic functions of NumPyThe np.arange() functionThe np.zeros() and np.ones() functionsThe np.linspace() functionOverview of PandasPandas data accessBoolean masking for filtering a DataFramePandas functions for exploring a DataFramePandas applying a functionThe Pandas groupby functionPandas multi-level indexingPandas pivot and melt functionsSummaryExercises
Technical requirementsDrawing the main plots in MatplotlibSummarizing numerical attributes using histograms or boxplotsObserving trends in the data using a line plotRelating two numerical attributes using a scatterplotModifying the visualsAdding a title to visuals and labels to the axisAdding legendsModifying ticksModifying markersSubplotsResizing visuals and saving themResizingSavingExample of Matplotilb assisting data preprocessingSummaryExercises
Technical requirementsWhat is data?Why this definition?DIKW pyramidData preprocessing for data analytics versus data preprocessing for machine learningThe most universal data structure – a tableData objectsData attributesTypes of data valuesAnalytics standpointProgramming standpointInformation versus patternUnderstanding everyday use of the word "information"Statistical use of the word "information"Statistical meaning of the word "pattern"SummaryExercisesReferences
Technical requirementsWhat is a database?Understanding the difference between a database and a datasetTypes of databasesThe differentiating elements of databasesRelational databases (SQL databases)Unstructured databases (NoSQL databases)A practical example that requires a combination of both structured and unstructured databasesDistributed databasesBlockchainConnecting to, and pulling data from, databasesDirect connectionWeb page connectionAPI connectionRequest connectionPublicly sharedSummaryExercises
Technical requirementsSummarizing a populationExample of summarizing numerical attributesExample of summarizing categorical attributesComparing populationsExample of comparing populations using boxplotsExample of comparing populations using histogramsExample of comparing populations using bar chartsInvestigating the relationship between two attributesVisualizing the relationship between two numerical attributesVisualizing the relationship between two categorical attributesVisualizing the relationship between a numerical attribute and a categorical attributeAdding visual dimensionsExample of a five-dimensional scatter plotShowing and comparing trendsExample of visualizing and comparing trendsSummaryExercise
Technical requirementsPredictive modelsForecastingRegression analysisLinear regressionExample of applying linear regression to perform regression analysisMLPHow does MLP work?Example of applying MLP to perform regression analysisSummaryExercises

Technical requirementsClassification modelsExample of designing a classification modelClassification algorithmsKNNExample of using KNN for classificationDecision TreesExample of using Decision Trees for classificationSummaryExercises
Technical requirementsClustering modelClustering example using a two-dimensional datasetClustering example using a three-dimensional datasetK-Means algorithmUsing K-Means to cluster a two-dimensional datasetUsing K-Means to cluster a dataset with more than two dimensionsCentroid analysisSummaryExercises
Technical requirementsThe levels, tools, and purposes of data cleaning – a roadmap to chapters 9, 10, and 11Purpose of data analyticsTools for data analyticsLevels of data cleaningMapping the purposes and tools of analytics to the levels of data cleaningData cleaning level I – cleaning up the tableExample 1 – unwise data collectionExample 2 – reindexing (multi-level indexing)Example 3 – intuitive but long column titlesSummaryExercises
Technical requirementsExample 1 – unpacking columns and reformulating the tableUnpacking FileNameUnpacking ContentReformulating a new table for visualizationThe last step – drawing the visualizationExample 2 – restructuring the tableExample 3 – level I and II data cleaningLevel I cleaningLevel II cleaningDoing the analytics – using linear regression to create a predictive modelSummaryExercises
Technical requirementsMissing valuesDetecting missing valuesExample of detecting missing valuesCauses of missing valuesTypes of missing valuesDiagnosis of missing valuesDealing with missing valuesOutliersDetecting outliersDealing with outliersErrors Types of errorsDealing with errorsDetecting systematic errorsSummaryExercises
Technical requirementsWhat are data fusion and data integration?Data fusion versus data integrationDirections of data integrationFrequent challenges regarding data fusion and integrationChallenge 1 – entity identificationChallenge 2 – unwise data collectionChallenge 3 – index mismatched formattingChallenge 4 – aggregation mismatchChallenge 5 – duplicate data objectsChallenge 6 – data redundancyExample 1 (challenges 3 and 4)Example 2 (challenges 2 and 3)Example 3 (challenges 1, 3, 5, and 6)Checking for duplicate data objectsDesigning the structure for the result of data integrationFilling songIntegrate_df from billboard_dfFilling songIntegrate_df from songAttribute_dfFilling songIntegrate_df from artist_dfChecking for data redundancyThe analysisExample summarySummary Exercise
Technical requirementsThe distinction between data reduction and data redundancyThe objectives of data reductionTypes of data reductionPerforming numerosity data reductionRandom samplingStratified samplingRandom over/undersamplingPerforming dimensionality data reductionLinear regression as a dimension reduction methodUsing a decision tree as a dimension reduction methodUsing random forest as a dimension reduction methodBrute-force computational dimension reductionPCAFunctional data analysisSummaryExercises
Technical requirementsThe whys of data transformation and massagingData transformation versus data massagingNormalization and standardizationBinary coding, ranking transformation, and discretizationExample one – binary coding of nominal attributeExample two – binary coding or ranking transformation of ordinal attributesExample three – discretization of numerical attributesUnderstanding the types of discretizationDiscretization – the number of cut-off pointsA summary – from numbers to categories and backAttribute constructionExample – construct one transformed attribute from two attributesFeature extractionExample – extract three attributes from one attributeExample – Morphological feature extractionFeature extraction examples from the previous chaptersLog transformationImplementation – doing it yourselfImplementation – the working module doing it for youSmoothing, aggregation, and binningSmoothingAggregationBinningSummaryExercise
Technical requirementsIntroducing the case studyThe audience of the results of analyticsIntroduction to the source of the dataIntegrating the data sourcesCleaning the dataDetecting and dealing with outliers and errorsDetecting and dealing with missing valuesAnalyzing the dataAnalysis question one – is there a significant difference between the mental health of employees across the attribute of gender?Analysis question two – is there a significant difference between the mental health of employees across the Age attribute?Analysis question three – do more supportive companies have mentally healthier employees?Analysis question four – does the attitude of individuals toward mental health influence their mental health and their seeking of treatments?Summary
Technical requirementsIntroducing the case studyIntroducing the source of the dataPreprocessing the dataDesigning the dataset to support the predictionFilling up the placeholder datasetSupervised dimension reductionAnalyzing the dataSummary
Technical requirementsIntroducing the case studyIntroduction to the source of the dataPreprocessing the dataTransforming election_df to partisan_dfCleaning edu_df, employ_df, pop_df, and pov_df Data integrationData cleaning level III – missing values, errors, and outliersChecking for data redundancyAnalyzing the dataUsing PCA to visualize the datasetK-Means clustering analysisSummary
A summary of the bookPart 1 – Technical requirementsPart 2 – Analytics goalsPart 3 – The preprocessingPart 4 – Case studiesPractice case studiesGoogle Covid-19 mobility datasetPolice killings in the USUS accidentsSan Francisco crimeData analytics job marketFIFA 2018 player of the matchHot hands in basketballWildfires in CaliforniaSilicon Valley diversity profileRecognizing fake job postingHunting more practice case studiesConclusions
Other Books You May EnjoyPackt is searching for authors like youShare Your Thoughts

Content preview from Hands-On Data Preprocessing in Python

Chapter 11: Data Cleaning Level III – Missing Values, Outliers, and Errors

In level I, we cleaned up the table without paying attention to the data structure or the recorded values. In level II, our attention was to have a data structure that would support our analytic goal, but we still didn't pay much attention to the correctness or appropriateness of the recorded values. That is the objective of data cleaning level III. In data cleaning level III, we will focus on the recorded values and will take measures to make sure that three matters regarding the values recorded in the data are addressed. First, we will make sure missing values in the data have been detected, that we know why this has happened, and that appropriate measures have been ...