book

Essential Statistics for Non-STEM Data Analysts

Name: Essential Statistics for Non-STEM Data Analysts
Author: Rongpeng Li
ISBN: 9781838984847

by Rongpeng Li

November 2020

Beginner

392 pages

8h 16m

English

Packt Publishing

Read now

Unlock full access

Essential Statistics for Non-STEM Data Analysts
Why subscribe?ContributorsAbout the authorAbout the reviewersPackt is searching for authors like you
Preface
Who this book is forWhat this book coversTo get the most out of this bookDownload the example code filesDownload the color imagesConventions usedGet in touchReviews
Section 1: Getting Started with Statistics for Data Science
Chapter 1: Fundamentals of Data Collection, Cleaning, and Preprocessing
Technical requirementsCollecting data from various data sourcesReading data directly from files Obtaining data from an APIObtaining data from scratchData imputationPreparing the dataset for imputationImputation with mean or median valuesImputation with the mode/most frequent valueOutlier removalData standardization – when and howExamples involving the scikit-learn preprocessing moduleImputationStandardizationSummary
Chapter 2: Essential Statistics for Data Assessment
Classifying numerical and categorical variablesDistinguishing between numerical and categorical variablesUnderstanding mean, median, and modeMeanMedianModeLearning about variance, standard deviation, quartiles, percentiles, and skewnessVarianceStandard deviationQuartilesSkewnessKnowing how to handle categorical variables and mixed data typesFrequencies and proportionsTransforming a continuous variable to a categorical oneUsing bivariate and multivariate descriptive statisticsCovarianceCross-tabulationSummary
Chapter 3: Visualization with Statistical Graphs
Basic examples with the Python Matplotlib packageElements of a statistical graphExploring important types of plotting in MatplotlibAdvanced visualization customizationCustomizing the geometryCustomizing the aestheticsQuery-oriented statistical plottingExample 1 – preparing data to fit the plotting function APIExample 2 – combining analysis with plain plottingPresentation-ready plotting tipsUse stylingFont matters a lotSummary
Section 2: Essentials of Statistical Analysis
Chapter 4: Sampling and Inferential Statistics
Understanding fundamental concepts in sampling techniquesPerforming proper sampling under different scenariosThe dangers associated with non-probability samplingProbability sampling – the safer approachUnderstanding statistics associated with samplingSampling distribution of the sample meanStandard error of the sample meanThe central limit theoremSummary
Chapter 5: Common Probability Distributions
Understanding important concepts in probabilityEvents and sample spaceThe probability mass function and the probability density functionSubjective probability and empirical probabilityUnderstanding common discrete probability distributionsBernoulli distributionBinomial distributionPoisson distributionUnderstanding the common continuous probability distribution Uniform distributionExponential distributionNormal distributionLearning about joint and conditional distributionIndependency and conditional distributionUnderstanding the power law and black swanThe ubiquitous power lawBe aware of the black swanSummary
Chapter 6: Parametric Estimation
Understanding the concepts of parameter estimation and the features of estimatorsEvaluation of estimatorsUsing the method of moments to estimate parametersExample 1 – the number of 911 phone calls in a dayExample 2 – the bounds of uniform distributionApplying the maximum likelihood approach with PythonLikelihood functionMLE for uniform distribution boundariesMLE for modeling noiseMLE and the Bayesian theoremSummary

Chapter 7: Statistical Hypothesis Testing
An overview of hypothesis testingUnderstanding P-values, test statistics, and significance levelsMaking sense of confidence intervals and P-values from visual examplesCalculating the P-value from discrete eventsCalculating the P-value from the continuous PDFSignificance levels in t-distributionThe power of a hypothesis testUsing SciPy for common hypothesis testing The paradigm T-testThe normality hypothesis testThe goodness-of-fit testA simple ANOVA modelStationarity tests for time seriesExamples of stationary and non-stationary time seriesAppreciating A/B testing with a real-world exampleConducting an A/B testRandomization and blockingCommon test statisticsCommon mistakes in A/B testsSummary
Section 3: Statistics for Machine Learning
Chapter 8: Statistics for Regression
Understanding a simple linear regression model and its rich contentLeast squared error linear regression and variance decompositionThe coefficient of determinationHypothesis testingConnecting the relationship between regression and estimatorsSimple linear regression as an estimatorHaving hands-on experience with multivariate linear regression and collinearity analysisCollinearityLearning regularization from logistic regression examplesSummary
Chapter 9: Statistics for Classification
Understanding how a logistic regression classifier worksThe formulation of a classification problemImplementing logistic regression from scratchEvaluating the performance of the logistic regression classifierBuilding a naïve Bayes classifier from scratchUnderfitting, overfitting, and cross-validationSummary
Chapter 10: Statistics for Tree-Based Methods
Overviewing tree-based methods for classification tasksGrowing and pruning a classification treeUnderstanding how splitting worksEvaluating decision tree performanceExploring regression treeUsing tree models in scikit-learnSummary
Chapter 11: Statistics for Ensemble Methods
Revisiting bias, variance, and memorizationUnderstanding the bootstrapping and bagging techniquesUnderstanding and using the boosting moduleExploring random forests with scikit-learnSummary
Section 4: Appendix
Chapter 12: A Collection of Best Practices
Understanding the importance of data qualityUnderstanding why data can be problematicAvoiding the use of misleading graphsExample 1 – COVID-19 trendExample 2 – Bar plot croppingFighting against false argumentsSummary
Chapter 13: Exercises and Projects
ExercisesChapter 1 – Fundamentals of Data Collection, Cleaning, and PreprocessingChapter 2 – Essential Statistics for Data AssessmentChapter 3 – Visualization with Statistical GraphsChapter 4 – Sampling and Inferential StatisticsChapter 5 – Common Probability DistributionsChapter 6 – Parameter EstimationChapter 7 – Statistical Hypothesis TestingChapter 8 – Statistics for RegressionChapter 9 – Statistics for ClassificationChapter 10 – Statistics for Tree-Based MethodsChapter 11 – Statistics for Ensemble MethodsProject suggestionsNon-tabular dataReal-time weather dataGoodness of fit for discrete distributionsBuilding a weather prediction web appBuilding a typing suggestion appFurther readingTextbooksVisualizationExercising your mindSummary
Other Books You May Enjoy
Leave a review - let other readers know what you think

Content preview from Essential Statistics for Non-STEM Data Analysts

Chapter 10: Statistics for Tree-Based Methods

In the previous chapter, we covered some important concepts in classification models. We also built a naïve Bayes classifier from scratch, which is very important because it requires you to understand every aspect of the details.

In this chapter, we are going to dive into another family of statistical models that are also widely used in statistical analysis as well as machine learning: tree-based models. Tree-based models can be used for both classification tasks and regression tasks.

By the end of this chapter, you will have achieved the following:

Gained an overview of tree-based classification
Understood the details of classification tree building
Understood the mechanisms of regression trees ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

APPLIED MULTIVARIATE STATISTICS: WITH SAS® SOFTWARE

Ravindra Khattree, Dayanand N. Naik

Practical Text Mining and Statistical Analysis for Non-structured Text Data Applications

Gary D. Miner, John Elder, Andrew Fast, Thomas Hill, Robert Nisbet, Dursun Delen

Performance Measurement in Finance

John Knight, Stephen Satchell

R: Data Analysis and Visualization

Tony Fischetti, Brett Lantz, Jaynal Abedin, Hrishi V. Mittal, Bater Makhabel, Edina Berlinger, Ferenc Illés, Milán Badics, Ádám Banai, Gergely Daróczi, Barbara Dömötör, Gergely Gabler, Dániel Havran, Péter Juhász, István Margitai, Balázs Márkus, Péter Medvegyev, Julia Molnár, Balázs Árpád Szucs, Ágnes Tuza, Tamás Vadász, Kata Váradi, Ágnes Vidovics-Dancs

Publisher Resources

ISBN: 9781838984847

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Essential Statistics for Non-STEM Data Analysts

by Rongpeng Li

Chapter 10: Statistics for Tree-Based Methods

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.