book

Julia for Data Science

Name: Julia for Data Science
Author: Anshul Joshi
ISBN: 9781785289699

by Anshul Joshi

September 2016

Beginner to intermediate

346 pages

6h 35m

English

Packt Publishing

Read now

Unlock full access

Julia for Data Science
Julia for Data Science
Credits
About the Author
About the Reviewer
www.PacktPub.com
Why subscribe?
Preface
What this book covers
What you need for this book
Who this book is for
Conventions

Reader feedback
Customer support
Downloading the example codeDownloading the color images of this bookErrataPiracyQuestions
1. The Groundwork – Julia's Environment
Julia is different
Setting up the environment
Installing Julia (Linux)Installing Julia (Mac)Installing Julia (Windows)Exploring the source code
Using REPL
Using Jupyter Notebook
Package management
Pkg.status() – package statusPkg.add() – adding packagesWorking with unregistered packagesPkg.update() – package updateMETADATA repositoryDeveloping packagesCreating a new package
Parallel computation using Julia
Julia's key feature – multiple dispatch
Methods in multiple dispatchAmbiguities – method definitions
Facilitating language interoperability
Calling Python code in Julia
Summary
References
2. Data Munging
What is data munging?The data munging process
What is a DataFrame?
The NA data type and its importanceDataArray – a series-like data structureDataFrames – tabular data structuresInstallation and using DataFrames.jlWriting the data to a fileWorking with DataFramesUnderstanding DataFrames joinsThe Split-Apply-Combine strategyReshaping the dataSorting a datasetFormula - a special data type for mathematical expressionsPooling dataWeb scraping
Summary
References
3. Data Exploration
SamplingPopulationWeight vectors
Inferring column types
Basic statistical summaries
Calculating the mean of the array or dataframe
Scalar statistics
Standard deviations and variances
Measures of variation
Z-scoresEntropyQuantilesModesSummary of datasets
Scatter matrix and covariance
Computing deviations
Rankings
Counting functions
Histograms
Correlation analysis
Summary
References
4. Deep Dive into Inferential Statistics
Installation
Understanding the sampling distribution
Understanding the normal distribution
Parameter estimation
Type hierarchy in Distributions.jl
Understanding SampleableRepresenting probabilistic distributions
Univariate distributions
Retrieving parametersStatistical functionsEvaluation of probabilitySampling in Univariate distributionsUnderstanding Discrete Univariate distributions and typesBernoulli distributionBinomial distributionContinuous distributionsCauchy distributionChi distributionChi-square distribution
Truncated distributions
Truncated normal distributions
Understanding multivariate distributions
Multinomial distributionMultivariate normal distributionDirichlet distribution
Understanding matrixvariate distributions
Wishart distributionInverse-Wishart distribution
Distribution fitting
Distribution selectionSymmetrical distributionsSkew distributions to the rightSkew distributions to the leftMaximum Likelihood EstimationSufficient statisticsMaximum-a-Posteriori estimation
Confidence interval
Interpreting the confidence intervalsUsage
Understanding z-score
Interpreting z-scores
Understanding the significance of the P-value
One-tailed and two-tailed test
Summary
References
5. Making Sense of Data Using Visualization
Difference between using and importall
Pyplot for Julia
Multimedia I/OInstallationBasic plottingPlot using sine and cosine
Unicode plots
InstallationExamplesGenerating Unicode scatterplotsGenerating Unicode line plots
Visualizing using Vega
InstallationExamplesScatterplotHeatmaps in Vega
Data visualization using Gadfly
Installing GadflyInteracting with Gadfly using plot functionExampleUsing Gadfly to plot DataFramesUsing Gadfly to visualize functions and expressionsGenerating an image with multiple layersGenerating plots with different aesthetics using statisticsThe step functionThe quantile-quantile functionTicks in GadflyGenerating plots with different aesthetics using GeometryBoxplotsUsing Geometry to create density plotsUsing Geometry to create histogramsBar plotsHistogram2d - the two-dimensional histogramSmooth line plotSubplot gridHorizontal and vertical linesPlotting a ribbonViolin plotsBeeswarm plotsElements - scalex_continuous and y_continuousx_discrete and y_discreteContinuous color scaleElements - guideUnderstanding how Gadfly works
Summary
References
6. Supervised Machine Learning
What is machine learning?Uses of machine learningMachine learning and ethics
Machine learning – the process
Different types of machine learningWhat is bias-variance trade-off?Effects of overfitting and underfitting on a model
Understanding decision trees
Building decision trees – divide and conquerWhere should we use decision tree learning?Advantages of decision treesDisadvantages of decision treesDecision tree learning algorithmsHow a decision tree algorithm worksUnderstanding and measuring purity of nodeAn example
Supervised learning using Naïve Bayes
Advantages of Naïve BayesDisadvantages of Naïve BayesUses of Naïve Bayes classificationHow Bayesian methods workPosterior probabilitiesClass-conditional probabilitiesPrior probabilitiesEvidenceThe bag of wordsAdvantages of using Naïve Bayes as a spam filterDisadvantages of Naïve Bayes filtersExamples of Naïve Bayes
Summary
References
7. Unsupervised Machine Learning
Understanding clusteringHow are clusters formed?Types of clusteringHierarchical clusteringOverlapping, exclusive, and fuzzy clusteringDifferences between partial versus complete clustering
K-means clustering
K-means algorithmAlgorithm of K-meansAssociating the data points with the closest centroidHow to choose the initial centroids?Time-space complexity of K-means algorithmsIssues with K-meansEmpty clusters in K-meansOutliers in the datasetDifferent types of clusterK-means – strengths and weaknessesBisecting K-means algorithmGetting deep into hierarchical clusteringAgglomerative hierarchical clusteringHow proximity is computedStrengths and weaknesses of hierarchical clusteringUnderstanding the DBSCAN techniqueSo, what is density?How are points classified using center-based densityDBSCAN algorithmStrengths and weaknesses of the DBSCAN algorithmCluster validationExample
Summary
References
8. Creating Ensemble Models
What is ensemble learning?Understanding ensemble learningHow to construct an ensembleCombination strategiesSubsampling training datasetBaggingWhen does bagging work?BoostingBoosting approachBoosting algorithmAdaBoost – boosting by samplingWhat is boosting doing?The bias and variance decompositionManipulating the input featuresInjecting randomness
Random forests
Features of random forestsHow do random forests work?The out-of-bag (oob) error estimateGini importanceProximities
Implementation in Julia
Learning and prediction
Why is ensemble learning superior?
Applications of ensemble learning
Summary
References
9. Time Series
What is forecasting?Decision-making processThe dynamics of a system
What is TimeSeries?
Trends, seasonality, cycles, and residualsDifference from standard linear regressionBasic objectives of the analysisTypes of modelsImportant characteristics to consider firstSystematic pattern and random noiseTwo general aspects of time series patternsTrend analysisSmoothingFitting a functionAnalysis of seasonalityAutocorrelation correlogramExamining correlogramsPartial autocorrelationsRemoving serial dependencyARIMACommon processesARIMA methodologyIdentificationEstimation and forecastingThe constant in ARIMA modelsIdentification phaseSeasonal modelsParameter estimationEvaluation of the modelInterrupted time series ARIMAExponential smoothingSimple exponential smoothingIndices of lack of fit (error)
Implementation in Julia
The TimeArray time series typeUsing time constraintswhenfromtofindwhenfindMathematical, comparison, and logical operatorsApplying methods to TimeSeriesLagLeadPercentageCombining methods in TimeSeriesMergeCollapseMap
Summary
References
10. Collaborative Filtering and Recommendation System
What is a recommendation system?The utility matrix
Association rule mining
Measures of association rulesHow to generate the item setsHow to generate the rules
Content-based filtering
Steps involved in content-based filteringAdvantages of content-based filteringLimitations of content-based filtering
Collaborative filtering
Baseline prediction methodsUser-based collaborative filteringItem-item collaborative filteringAlgorithm of item-based collaborative filtering
Building a movie recommender system
Summary
11. Introduction to Deep Learning
Revisiting linear algebraA gist of scalarsA brief outline of vectorsThe importance of matricesWhat are tensors?
Probability and information theory
Why probability?
Differences between machine learning and deep learning
What is deep learning?Deep feedforward networksUnderstanding the hidden layers in a neural networkThe motivation of neural networksUnderstanding regularizationOptimizing deep learning modelsThe case of optimization
Implementation in Julia
Network architectureTypes of layersNeurons (activation functions)Understanding regularizers for ANNNorm constraintsUsing solvers in deep neural networksCoffee breaksImage classification with pre-trained Imagenet CNN
Summary
References

Content preview from Julia for Data Science

Scatter matrix and covariance

Covariance is used very often by data scientists to find out how two ordered sets of data follow in the same direction. It can very easily define whether the variables are correlated or not. To best represent this behavior, we create a covariance matrix. The unnormalized version of the covariance matrix is the scatter matrix.

To create a scatter matrix, we use the scattermat(arr) function.

The default behavior is to treat each row as an observation and column as a variable. This can be changed by providing the keyword arguments vardim and mean:

Vardim: vardim=1 (default) means each column is a variable and each row is an observation. vardim=2 is the reverse.
mean: The mean is computed by scattermat. We can use a predefined ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781785289699

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Julia for Data Science

by Anshul Joshi

Scatter matrix and covariance

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.