book

Practical Data Science with Python

by Nathan George

September 2021

Beginner to intermediate

620 pages

15h 30m

English

Packt Publishing

Read now

Unlock full access

Who this book is forWhat this book coversTo get the most out of this bookGet in touch
The data science origin storyThe top data science tools and skillsPythonOther programming languagesGUIs and platformsCloud toolsStatistical methods and mathCollecting, organizing, and preparing dataSoftware developmentBusiness understanding and communicationSpecializations in and around data scienceMachine learningBusiness intelligenceDeep learningData engineeringBig dataStatistical methodsNatural Language Processing (NLP)Artificial Intelligence (AI)Choosing how to specializeData science project methodologiesUsing data science in other fieldsCRISP-DMTDSPFurther reading on data science project management strategiesOther toolsTest your knowledgeSummary
Installing Python with Anaconda and getting startedInstalling AnacondaRunning Python codeThe Python shellThe IPython shellJupyterWhy the command line?Command line basicsInstalling and using a code text editor – VS CodeEditing Python code with VS CodeRunning a Python fileInstalling Python packages and creating virtual environmentsPython basicsNumbersStringsVariablesLists, tuples, sets, and dictionariesListsTuplesSetsDictionariesLoops and comprehensionsBooleans and conditionalsPackages and modulesFunctionsClassesMultithreading and multiprocessingSoftware engineering best practicesDebugging errors and utilizing documentationDebuggingDocumentationVersion control with GitCode styleProductivity tipsTest your knowledgeSummary
IntroductionLoading, reading, and writing files with base PythonOpening a file and reading its contentsUsing the built-in JSON moduleSaving credentials or data in a Python fileSaving Python objects with pickleUsing SQLite and SQLCreating a SQLite database and storing dataUsing the SQLAlchemy package in PythonTest your knowledgeSummary
Data wrangling and analyzing iTunes dataLoading and saving data with PandasUnderstanding the DataFrame structure and combining/concatenating multiple DataFramesExploratory Data Analysis (EDA) and basic data cleaning with PandasExamining the top and bottom of the dataExamining the data's dimensions, datatypes, and missing valuesInvestigating statistical properties of the dataPlotting with DataFramesCleaning dataFiltering DataFramesRemoving irrelevant dataDealing with missing valuesDealing with outliersDealing with duplicate valuesEnsuring datatypes are correctStandardizing data formatsData transformationsUsing replace, map, and apply to clean and transform dataUsing GroupByWriting DataFrames to diskWrangling and analyzing Bitcoin price dataUnderstanding NumPy basicsUsing NumPy mathematical functionsTest your knowledgeSummary
EDA and visualization libraries in PythonPerforming EDA with Seaborn and pandasMaking boxplots and letter-value plotsMaking histograms and violin plotsMaking scatter plots with Matplotlib and SeabornExamining correlations and making correlogramsMaking missing value plotsUsing EDA Python packagesUsing visualization best practicesSaving plots for sharing and reportsMaking plots with PlotlyTest your knowledgeSummary
Parsing and processing Word and PDF documentsReading text from Word documentsExtracting insights from Word documents: common words and phrasesAnalyzing words and phrases from the textReading text from PDFsReading and writing data with Excel filesUsing pandas for wrangling Excel filesAnalyzing the dataUsing openpyxl for wrangling Excel filesTest your knowledgeSummary
Understanding the structure of the internetGET and POST requests, and HTMLPerforming simple web scrapingUsing urllibUsing the requests packageScraping several filesExtracting the data from the scraped filesParsing HTML from scraped pagesUsing XPath, lxml, and bs4 to extract data from webpagesCollecting data from several pagesUsing APIs to collect dataUsing API wrappersThe ethics and legality of web scrapingTest your knowledgeSummary

Probability basicsIndependent and conditional probabilitiesBayes' TheoremFrequentist versus BayesianDistributionsThe normal distribution and using scipy to generate distributionsDescriptive statistics of distributionsVariants of the normal distributionFitting distributions to data to get parametersThe Student's t-distributionThe Bernoulli distributionThe binomial distributionThe uniform distributionThe exponential and Poisson distributionsThe Weibull distributionThe Zipfian distributionSampling from dataThe law of large numbersThe central limit theoremRandom samplingBootstrap sampling and confidence intervalsTest your knowledgeSummary
Statistical testing basics and sample comparison testsThe t-test and z-testOne-sample, two-sided t-testThe z-testOne-sided testsTwo-sample t- and z-tests: A/B testingPaired t- and z-testsOther A/B testing methodsTesting between several groups with ANOVAPost-hoc ANOVA testsAssumptions for these methodsOther statistical testsTesting if data belongs to a distributionGeneralized ESD outlier testThe Pearson correlation testTest your knowledgeSummary
Types of machine learningFeature selectionThe curse of dimensionalityOverfitting and underfitting, and the bias-variance trade-offMethods for feature selectionVariance thresholding – removing features with too much and too little varianceUnivariate statistics feature selectionCorrelationMutual information score and chi-squaredThe chi-squared testANOVAUsing the univariate statistics for feature selectionFeature engineeringData cleaning and preparationConverting strings to datesOutlier cleaning strategiesCombining multiple columnsTransforming numeric dataStandardizationMaking data more Gaussian with the Yeo-Johnson transformExtracting datetime featuresBinningOne-hot encoding and label encodingSimplifying categorical columnsOne-hot encodingDimensionality reductionPrinciple Component Analysis (PCA)Test your knowledgeSummary
Machine learning classification algorithmsLogistic regression for binary classificationGetting predictions from our modelHow logistic regression worksOdds ratio and the logitExamining feature importances with sklearnUsing statmodels for logistic regressionMaximum likelihood estimation, optimizers, and the logistic regression algorithmRegularizationHyperparameters and cross-validationLogistic regression (and other models) with big dataNaïve Bayes for binary classificationk-nearest neighbors (KNN)Multiclass classificationLogistic regressionOne-versus-rest and one-versus-one formulationsMulti-label classificationChoosing a model to useThe "no free lunch" theoremComputational complexity of modelsTest your knowledgeSummary
Evaluating classification algorithm performance with metricsTrain-validation-test splitsAccuracyCohen's KappaConfusion matrixPrecision, recall, and F1 scoreAUC score and the ROC curveChoosing the optimal cutoff threshold Sampling and balancing classification dataDownsamplingOversamplingSMOTE and other synthetic sampling methodsTest your knowledgeSummary
Linear regressionLinear regression with sklearnLinear regression with statsmodelsRegularized linear regressionRegression with KNN in sklearnEvaluating regression modelsR2 or the coefficient of determinationAdjusted R2Information criteriaMean squared errorMean absolute errorLinear regression assumptionsRegression models on big dataForecastingTest your knowledgeSummary
Hyperparameter optimization with search methodsUsing grid searchUsing random searchUsing Bayesian searchOther advanced search methodsUsing learning curvesOptimizing the number of features with ML modelsUsing AutoML with PyCaretThe no free lunch theoremAutoML solutionsUsing PyCaretTest your knowledgeSummary
Decision treesRandom forestsRandom forests with sklearnRandom forests with H2OFeature importance from tree-based methodsUsing H2O for feature importanceUsing sklearn random forest feature importancesBoosted trees: AdaBoost, XGboost, LightGBM, and CatBoostAdaBoostXGBoostXGBoost with PyCaretXGBoost with the xgboost packageTraining boosted models on a GPULightGBMLightGBM plottingUsing LightGBM directlyCatBoostUsing CatBoost nativelyUsing early stopping with boosting algorithmsTest your knowledgeSummary
How SVMs workSVMs for classificationSVMs for regressionUsing SVMsUsing SVMs in sklearnTuning SVMs with pycaretTest your knowledgeSummary
Using k-means clusteringClustering metricsOptimizing k in k-meansExamining the clustersHierarchical clusteringDBSCANOther unsupervised methodsTest your knowledgeSummary
Text preprocessingBasic text cleaningStemming and LemmatizingPreparing text with spaCyWord vectorsTFIDF vectorsBasic text analysisWord frequency plotsWordcloudsZipf's lawWord collocationsParts of speechUnsupervised learningTopic modelingTopic modeling with pycaretTopic modeling with Top2VecSupervised learningClassificationSentiment analysisTest your knowledgeSummary
Data storytellingData storytelling exampleAutomated reporting and dashboardingAutomated reporting optionsAutomated dashboardingScheduling tasks to run automatically Test your knowledgeSummary
The ethics of machine learning algorithmsBiasHow to decrease ML biasesCarefully evaluating performance and consequencesData privacyData privacy regulations and lawsk-anonymity, l-diversity, and t-closenessDifferential privacyUsing data science for the common goodOther ethical considerationsTest your knowledgeSummary
Blogs, newsletters, books, and academic sourcesBlogsNewslettersBooksAcademic sourcesData science competition websitesOnline learning platformsCloud servicesOther places to keep an eye onStrategies for staying up to dateOther data science topics we didn't coverThe future of data scienceSummary

Content preview from Practical Data Science with Python

9 Statistical Testing for Data Science

In the previous chapter, we laid the groundwork for understanding probability and statistics. Now, we will leverage that understanding to perform statistical tests that we can use to test hypotheses. We will cover the following statistical tests in this chapter:

The t-test, z-test, and bootstrapping for comparing the means of data (for example, A/B testing)
The ANOVA test for comparing the means of groups
Testing if data comes from a distribution (for example, a Gaussian distribution)
Testing for outliers with the scikit-posthocs package
Tests for relationships between variables (Pearson and chi-squared tests)

This is only a small number of the total amount of statistical tests out there, but there are ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Start your free trial

Data Science Projects with Python - Second Edition

Stephen Klosterman

Python: End-to-end Data Analysis

Phuong Vothihong, Martin Czygan, Ivan Idris, Magnus Vilhelm Persson, Luiz Felipe Martins

Python for Data Science

Yuli Vasiliev

Interpretable Machine Learning with Python

Serg Masís

Publisher Resources

ISBN: 9781801071970Supplemental Content

9

Statistical Testing for Data Science