book

R: Data Analysis and Visualization

by Tony Fischetti, Brett Lantz, Jaynal Abedin, Hrishi V. Mittal, Bater Makhabel, Edina Berlinger, Ferenc Illés, Milán Badics, Ádám Banai, Gergely Daróczi, Barbara Dömötör, Gergely Gabler, Dániel Havran, Péter Juhász, István Margitai, Balázs Márkus, Péter Medvegyev, Julia Molnár, Balázs Árpád Szucs, Ágnes Tuza, Tamás Vadász, Kata Váradi, Ágnes Vidovics-Dancs

June 2016

Beginner to intermediate

1783 pages

71h 22m

English

Packt Publishing

Read now

Unlock full access

Meet Your Course Guide
Navigating the basicsArithmetic and assignmentLogicals and charactersFlow of control
SubsettingVectorized functionsAdvanced subsettingRecycling

Univariate data
Multivariate data
CovarianceCorrelation coefficientsComparing multiple correlations
Categorical and continuous variablesTwo categorical variablesTwo continuous variablesMore than two continuous variables
Basic probability
ParametersThe binomial distribution
The three-sigma rule and using z-tables
Estimating means
How did we get 1.96?
Null Hypothesis Significance TestingOne and two-tailed testsWhen things go wrongA warning about significanceA warning about p-values
Assumptions of the one sample t-test
Don't be fooled!Assumptions of the independent samples t-test
Assumptions of ANOVA
The big idea behind Bayesian analysis
Linear models
A word of warning
Cross-validationStriking a balance
Second Anscombe relationshipThird Anscombe relationshipFourth Anscombe relationship
k-Nearest NeighborsUsing k-NN in RConfusion matricesLimitations of k-NN
Using logistic regression in R
The vertical decision boundaryThe diagonal decision boundaryThe crescent decision boundaryThe circular decision boundary
Relational DatabasesWhy didn't we just do that in SQL?
Analysis with missing dataVisualizing missing dataTypes of missing dataSo which one is it?Unsophisticated methods for dealing with missing dataComplete case analysisPairwise deletionMean substitutionHot deck imputationRegression imputationStochastic regression imputationMultiple imputationSo how does mice come up with the imputed values?Methods of imputationMultiple imputation in practice
Checking for out-of-bounds dataChecking the data type of a columnChecking for unexpected categoriesChecking for outliers, entry errors, or unlikely data pointsChaining assertions
OpenRefineRegular expressionstidyr
Wait to optimize
Allocation of memoryVectorization
Getting started with parallel RAn example of (some) substance
R ScriptingRStudioRunning R scriptsAn example scriptScripting and reproducibility
Base graphics using the default package
Introduction
Getting readyHow to do it...How it works...There's more...A note on R's built-in datasetsSee also
Getting readyHow to do it...How it works...There's more...See also
Getting readyHow to do it...How it works...There's more...See also
How to do it...How it works...There's more...See also
Getting readyHow to do it...How it works...There's more...See also
How to do it...How it works...There's more...See also
How to do it...How it works...There's more...See also
How to do it...How it works...There's more...See also
How to do it...How it works...There's more...See also
Getting readyHow to do it...How it works...There's more...See also
Getting readyHow to do it...How it works...There's more...See also
How to do it...How it works...There's more...See also
Introduction
Getting readyHow to do it...How it works...There's more...See also
Getting readyHow to do it...How it works...There's more...
Getting readyHow to do it...How it works...There's more...
Getting readyHow to do it...How it works...There's more...See also
Getting readyHow to do it...How it works...There's more...See also
Getting readyHow to do it...How it works...There's more...See also
Getting readyHow to do it...How it works...See also
Getting readyHow to do it...How it works...There's more...
Getting readyHow to do it...How it works...There's more...See also
Getting readyHow to do it...How it works...There's more...
Getting readyHow to do it...How it works...See also
Introduction
Getting readyHow to do it...How it works...There's more...See also
Getting readyHow to do it...How it works...
Getting readyHow to do it...How it works...There's more...
Getting readyHow to do it...How it works...
Getting readyHow to do it...How it works...There's more...
Getting readyHow to do it...How it works...
Getting readyHow to do it...How it works...
Getting readyHow to do it...How it works...
Getting readyHow to do it...How it works...
Getting readyHow to do it...How it works...There's more...
Getting readyHow to do it...How it works...There's more...
Getting readyHow to do it...How it works...There's more...
Getting readyHow to do it...How it works...There's more...
Introduction
Getting readyHow to do it...How it works...There's more...See also
Getting readyHow to do it...How it works...There's more...
Getting readyHow to do it...How it works...There's more...See also
Getting readyHow to do it...How it works...There's more...
Getting readyHow to do it...How it works...
Getting readyHow to do it...How it works...There's more...
Getting readyHow to do it...How it works...There's more...
Getting readyHow to do it...How it works...There's more...
Getting readyHow to do it...How it works...There's more...
Getting readyHow to do it...How it works...There's more...
Getting readyHow to do it...How it works...
Getting readyHow to do it...How it works...There's more...
Introduction
Getting readyHow to do it...How it works...See also
Getting readyHow to do it...How it works...There's more...
Getting readyHow to do it...How it works...There's more...
Getting readyHow to do it...How it works...There's more...
Getting readyHow to do it...How it works...There's more...See also
Getting readyHow to do it...How it works...There's more...
Getting readyHow to do it...How it works...There's more...
Getting readyHow to do it...How it works...
Getting readyHow to do it...How it works...See also
Getting readyHow it works...There's more...See also
Getting readyHow to do it...How it works...There's more...
Introduction
Getting readyHow to do it...How it works...There's more
Getting readyHow to do it...How it works...There's more
Getting readyHow to do it...How it works...There's more
Getting readyHow to do it...How it works...
Getting readyHow to do it...How it works...
Getting readyHow to do it...How it works...
Introduction
Getting readyHow to do it...How it works...There's moreSee also
Getting readyHow to do it...How it works...There's moreSee also
Getting readyHow to do it...How it works...
Getting readyHow to do it...How it works...There's more
Getting readyHow to do it...How it works...See also
Getting readyHow to do it...How it works...
Getting readyHow to do it...How it works...There's more
Getting readyHow to do it...How it works...There's more
Getting readyHow to do it...How it works...There's more
Getting readyHow to do it...How it works...There's more
Introduction
Getting readyHow to do it...How it works...There's moreSee also
Getting readyHow to do it...How it works...There's more
Getting readyHow to do it...How it works...There's more
Getting readyHow to do it...How it works...There's moreSee also
Getting readyHow to do it...How it works...There's moreSee also
Getting readyHow to do it...How it works...There's more
Getting readyHow to do it...How it works...There's more
Introduction
Getting readyHow to do it...How it works...There's moreSee also
Getting readyHow to do it...How it works...There's more
Getting readyHow to do it...How it works...There's moreSee also
Getting readyHow to do it...How it works...See Also
Getting readyHow to do it...How it works...There's more
Introduction
Getting readyHow to do it…How it works…There's more…See also
Getting readyHow to do it…How it works…There's more…See also
Getting readyHow to do it…How it works…There's more…
Getting readyHow to do it…How it works…There's more…See also
Getting readyHow to do it…How it works…There's more…
Getting readyHow to do it…How it works…There's more…
Getting readyHow to do it…How it works…There's more…
Getting readyHow to do it…How it works…There's more…
Getting readyHow to do it…How it works…There's more…
Introduction
Getting readyHow to do it…How it works…There's more…See also
Getting readyHow to do it…How it works…There's more…See also
Getting readyHow to do it…How it works…There's more…
Getting readyHow to do it...How it works…There's more...
Getting readyHow to do it...How it works…
Getting readyHow to do it...How it works…There's more...
Getting readyHow to do it...How it works…There's more...
Getting readyHow to do it...How it works...
Introduction
Getting readyHow to do it…How it works…There's more…See also
Getting readyHow to do it…How it works…There's more…
Getting readyHow to do it…
Getting readyHow to do it...How it works…There's more...
Introduction
Getting readyHow to do it…How it works…There's more…See also...
Getting readyHow to do it…How it works…There's more…
Getting readyHow to do it…How it works…
Getting readyHow to do it...How it works…
Introduction
Getting readyHow to do it...How it works...There's moreSee also
Getting readyHow to do it...How it works...There's more
Getting readyHow to do it...How it works...There's more
Getting readyHow to do it...How it works...There's more
Getting readyHow to do it...How it works...There's more
Getting readyHow to do it...How it works...There's moreSee also
Getting readyHow to do it...How it works...There's more
Big dataScalability and efficiency
Feature extractionSummarizationThe data mining processCRISP-DMSEMMA
Social network
Information retrieval and text miningMining text for prediction
What are the disadvantages of R?
Statistics and data miningStatistics and machine learningStatistics and RThe limitations of statistics on data mining
Approaches to machine learningMachine learning architecture
Numeric attributesCategorical attributesData descriptionData measuring
Missing valuesJunk, noisy data, or outlier
Eigenvalues and EigenvectorsPrincipal-Component AnalysisSingular-value decompositionCUR decomposition
Data transformationNormalization data transformation methodsData discretization
Visualization with R
An overview of associations and patternsPatterns and pattern discoveryThe frequent itemsetThe frequent subsequenceThe frequent substructuresRelationship or rules discoveryAssociation rulesCorrelation rules
The market basket modelA-Priori algorithmsInput data characteristics and data structureThe A-Priori algorithmThe R implementationA-Priori algorithm variantsThe Eclat algorithmThe R implementationThe FP-growth algorithmInput data characteristics and data structureThe FP-growth algorithmThe R implementationThe GenMax algorithm with maximal frequent itemsetsThe R implementationThe Charm algorithm with closed frequent itemsetsThe R implementationThe algorithm to generate association rulesThe R implementation
Mining multilevel and multidimensional association rulesConstraint-based frequent pattern mining
Sequence datasetThe GSP algorithm
The SPADE algorithmThe R implementationRule generation from sequential patterns
Classification
Attribute selection measuresTree pruningGeneral algorithm for the decision tree generationThe R implementation
The ID3 algorithmThe R implementationWeb attack detectionHigh-value credit card customers classification
The C4.5 algorithmThe R implementationA parallel version with MapReduceWeb spam detection
The CART algorithmThe R implementationWeb key resource page judgment
EstimatingPrior probability estimationLikelihood estimationThe Bayes classificationThe R implementationTrojan traffic identification method
The Naïve Bayes classificationThe R implementationIdentify spam e-mail
Transformation from decision tree to decision rulesRule-based classificationSequential covering algorithmThe RIPPER algorithmThe R implementationRule-based classification of player types in computer games
Ensemble (EM) methodsThe bagging algorithmThe boosting and AdaBoost algorithmsThe Random forests algorithmThe R implementationParallel version with MapReduce
The Bayesian belief network (BBN) algorithmThe R implementationBiological traits
The kNN algorithmThe R implementation
The SVM algorithmThe R implementationParallel version with MapReduceDocument retrieval
The associative classificationCBADiscriminative frequent pattern-based classificationThe R implementationText classification using sentential frequent itemsets
The BP algorithmThe R implementationParallel version with MapReduce
Search engines and the k-means algorithmThe k-means clustering algorithmThe kernel k-means algorithmThe k-modes algorithmThe R implementationParallel version with MapReduceSearch engine and web page clustering
The PAM algorithmThe R implementationAutomatic abstraction and summarization of document text
The CLARA algorithmThe R implementation
The CLARANS algorithmThe R implementation
Affinity propagation clusteringThe R implementationUnsupervised image categorizationThe spectral clustering algorithmThe R implementation
Agglomerative hierarchical clusteringThe BIRCH algorithmThe chameleon algorithmThe Bayesian hierarchical clustering algorithmThe probabilistic hierarchical clustering algorithmThe R implementationNews categorization
Customer categorization analysis of e-commerce and DBSCANThe DBSCAN algorithmCustomer categorization analysis of e-commerce
The OPTICS algorithmThe R implementationClustering web pages
The DENCLUE algorithmThe R implementationVisitor analysis in the browser cache
The STING algorithmThe R implementationRecommendation systems
The CLIQUE algorithmThe R implementationWeb sentiment analysis
The WAVE cluster algorithmThe R implementationOpinion mining
The EM algorithmThe R implementationThe user search intent
The MAFIA algorithmThe SURFING algorithmThe R implementationCustomer purchase data analysis
The SCAN algorithmThe R implementationSocial networking service (SNS)
Credit card fraud detection and statistical methodsThe likelihood-based outlier detection algorithmThe R implementationCredit card fraud detection
The NL algorithmThe FindAllOutsM algorithmThe FindAllOutsD algorithmThe distance-based algorithmThe Dolphin algorithmThe R implementationActivity monitoring and the detection of mobile fraud
The OPTICS-OF algorithmThe High Contrast Subspace algorithmThe R implementationIntrusion detection
Hierarchical clustering to detect outliersThe k-means-based algorithmThe ODIN algorithmThe R implementation
The OCSVM algorithmThe one-class nearest neighbor algorithmThe R implementationMonitoring the performance of the web server
The conditional anomaly detection (CAD) algorithmThe R implementationDetecting novelty in text and topic detection
The route outlier detection (ROD) algorithmThe R implementationCharacteristics of collective outliers
The brute-force algorithmThe HilOut algorithmThe R implementation
The credit card transaction flow and STREAM algorithmThe STREAM algorithmThe single-pass-any-time clustering algorithmThe R implementationThe credit card transaction flow
The ARIMA algorithmPredicting future prices
The hError algorithmTime-series classification with the 1NN classifierThe R implementationStock market data
The TECNO-STREAMS algorithmThe R implementationWeb click streams
The PrefixSpan algorithmThe R implementation
Graph miningGraphGraph mining algorithms
The gPLS algorithmThe GraphSig algorithmThe gSpan algorithmRightmost path extensions and their supportsThe subgraph isomorphism enumeration algorithmThe canonical checking algorithmThe R implementation
Community detection and the shingling algorithmThe node classification and iterative classification algorithmsThe R implementation
Text mining and TM packages
Topic representationThe multidocument summarization algorithmThe Maximal Marginal Relevance algorithmThe R implementation
The N-gram-based text categorizationThe R implementation
The FCA-based association rule mining algorithmThe R implementation
Multivariate time series analysisCointegrationVector autoregressive modelsVAR implementation exampleCointegrated VAR and VECM
GARCH modeling with the rugarch packageThe standard GARCH modelThe Exponential GARCH model (EGARCH)The Threshold GARCH model (TGARCH)Simulation and forecasting
Arbitrage pricing theoryImplementation of APTFama-French three-factor model
Data selectionEstimation of APT with principal component analysisEstimation of the Fama-French model
Motivation
The dataLoading the dataThe seasonal componentAR(1) estimation and forecastingSETAR estimation and forecastingInterpreting the resultsReferences
Getting data from open sources
Loading big matricesBig data K-means clustering analysis
Loading big dataFitting a linear regression model on large datasets
Terminology and notations
Two-dimensional Wiener processesThe Margrabe formulaApplication in R
Pricing formula for a call quantoPricing a call quanto in R
The Black modelPricing a cap with Black's model
A general pricing approach
Hedging of derivativesMarket risk of derivativesStatic delta hedgeDynamic delta hedgeComparing the performance of delta hedging
Optimization of the hedgeOptimal hedging in the case of absolute transaction costsOptimal hedging in the case of relative transaction costs
The basics of fundamental analysis
Market efficiency
The TA toolkitMarketsPlotting charts - bitcoinBuilt-in indicatorsSMA and EMARSIMACDCandle patterns: key reversalEvaluating the signals and managing the positionA word on money managementWraping up
Forecasting bitcoin pricesEvaluation of the strategy
A universally consistent, non-parametric investment strategyEvaluation of the strategy
Data preparationData source at first glanceCash-flow generator functionsPreparing the cash-flow
A Model of deposit interest rate developmentStatic replication of non-maturity deposits
Principles of the Basel AccordsBasel IBasel IIMinimum capital requirementsSupervisory reviewTransparencyBasel III
Analytical VaRHistorical VaRMonte-Carlo simulation
Market riskCredit riskOperational risk
Systemic risk in a nutshell
Implementation in RResults
The simulationImplementation in RResults
The origins of machine learning
Machine learning successesThe limits of machine learningMachine learning ethics
Data storageAbstractionGeneralizationEvaluation
Types of input dataTypes of machine learning algorithmsMatching input data to algorithms
Installing R packagesLoading and unloading R packages
R data structuresVectorsFactorsListsData framesMatrixes and arrays
Saving, loading, and removing R data structuresImporting and saving data from CSV files
Exploring the structure of dataExploring numeric variablesMeasuring the central tendency – mean and medianMeasuring spread – quartiles and the five-number summaryVisualizing numeric variables – boxplotsVisualizing numeric variables – histogramsUnderstanding numeric data – uniform and normal distributionsMeasuring spread – variance and standard deviationExploring categorical variablesMeasuring the central tendency – the modeExploring relationships between variablesVisualizing relationships – scatterplotsExamining relationships – two-way cross-tabulations
Understanding nearest neighbor classificationThe k-NN algorithmMeasuring similarity with distanceChoosing an appropriate kPreparing data for use with k-NNWhy is the k-NN algorithm lazy?
Step 1 – collecting dataStep 2 – exploring and preparing the dataTransformation – normalizing numeric dataData preparation – creating training and test datasetsStep 3 – training a model on the dataStep 4 – evaluating model performanceStep 5 – improving model performanceTransformation – z-score standardizationTesting alternative values of k
Understanding Naive BayesBasic concepts of Bayesian methodsUnderstanding probabilityUnderstanding joint probabilityComputing conditional probability with Bayes' theoremThe Naive Bayes algorithmClassification with Naive BayesThe Laplace estimatorUsing numeric features with Naive Bayes
Step 1 – collecting dataStep 2 – exploring and preparing the dataData preparation – cleaning and standardizing text dataData preparation – splitting text documents into wordsData preparation – creating training and test datasetsVisualizing text data – word cloudsData preparation – creating indicator features for frequent wordsStep 3 – training a model on the dataStep 4 – evaluating model performanceStep 5 – improving model performance
Understanding decision treesDivide and conquerThe C5.0 decision tree algorithmChoosing the best splitPruning the decision tree
Step 1 – collecting dataStep 2 – exploring and preparing the dataData preparation – creating random training and test datasetsStep 3 – training a model on the dataStep 4 – evaluating model performanceStep 5 – improving model performanceBoosting the accuracy of decision treesMaking mistakes more costlier than others
Separate and conquerThe 1R algorithmThe RIPPER algorithmRules from decision treesWhat makes trees and rules greedy?
Step 1 – collecting dataStep 2 – exploring and preparing the dataStep 3 – training a model on the dataStep 4 – evaluating model performanceStep 5 – improving model performance
Understanding regressionSimple linear regressionOrdinary least squares estimationCorrelationsMultiple linear regression
Step 1 – collecting dataStep 2 – exploring and preparing the dataExploring relationships among features – the correlation matrixVisualizing relationships among features – the scatterplot matrixStep 3 – training a model on the dataStep 4 – evaluating model performanceStep 5 – improving model performanceModel specification – adding non-linear relationshipsTransformation – converting a numeric variable to a binary indicatorModel specification – adding interaction effectsPutting it all together – an improved regression model
Adding regression to trees
Step 1 – collecting dataStep 2 – exploring and preparing the dataStep 3 – training a model on the dataVisualizing decision treesStep 4 – evaluating model performanceMeasuring performance with the mean absolute errorStep 5 – improving model performance
Understanding neural networksFrom biological to artificial neuronsActivation functionsNetwork topologyThe number of layersThe direction of information travelThe number of nodes in each layerTraining neural networks with backpropagation
Step 1 – collecting dataStep 2 – exploring and preparing the dataStep 3 – training a model on the dataStep 4 – evaluating model performanceStep 5 – improving model performance
Classification with hyperplanesThe case of linearly separable dataThe case of nonlinearly separable dataUsing kernels for non-linear spaces
Step 1 – collecting dataStep 2 – exploring and preparing the dataStep 3 – training a model on the dataStep 4 – evaluating model performanceStep 5 – improving model performance
Understanding association rulesThe Apriori algorithm for association rule learningMeasuring rule interest – support and confidenceBuilding a set of rules with the Apriori principle
Step 1 – collecting dataStep 2 – exploring and preparing the dataData preparation – creating a sparse matrix for transaction dataVisualizing item support – item frequency plotsVisualizing the transaction data – plotting the sparse matrixStep 3 – training a model on the dataStep 4 – evaluating model performanceStep 5 – improving model performanceSorting the set of association rulesTaking subsets of association rulesSaving association rules to a file or data frame
Understanding clusteringClustering as a machine learning taskThe k-means clustering algorithmUsing distance to assign and update clustersChoosing the appropriate number of clusters
Step 1 – collecting dataStep 2 – exploring and preparing the dataData preparation – dummy coding missing valuesData preparation – imputing the missing valuesStep 3 – training a model on the dataStep 4 – evaluating model performanceStep 5 – improving model performance
Measuring performance for classificationWorking with classification prediction data in RA closer look at confusion matricesUsing confusion matrices to measure performanceBeyond accuracy – other measures of performanceThe kappa statisticSensitivity and specificityPrecision and recallThe F-measureVisualizing performance trade-offsROC curves
The holdout methodCross-validationBootstrap sampling
Tuning stock models for better performanceUsing caret for automated parameter tuningCreating a simple tuned modelCustomizing the tuning process
Understanding ensemblesBaggingBoostingRandom forestsTraining random forestsEvaluating random forest performance
Working with proprietary files and databasesReading from and writing to Microsoft Excel, SAS, SPSS, and Stata filesQuerying data in SQL databases
Downloading the complete text of web pagesScraping data from web pagesParsing XML documentsParsing JSON from web APIs
Analyzing bioinformatics dataAnalyzing and visualizing network data
Managing very large datasetsGeneralizing tabular data structures with dplyrMaking data frames faster with data.tableCreating disk-based data frames with ffUsing massive matrices with bigmemoryLearning faster with parallel computingMeasuring execution timeWorking in parallel with multicore and snowTaking advantage of parallel with foreach and doParallelParallel cloud computing with MapReduce and HadoopGPU computingDeploying optimized learning algorithmsBuilding bigger regression models with biglmGrowing bigger and faster random forests with bigrfTraining and evaluating models in parallel with caret
Module 1: Data Analysis with RChapter 1: RefresheRChapter 2: The Shape of DataChapter 3: Describing RelationshipsChapter 4: ProbabilityChapter 5: Using Data to Reason About the WorldChapter 6: Testing HypothesesChapter 7: Bayesian MethodsChapter 8: Predicting Continuous VariablesChapter 9: Predicting Categorical VariablesChapter 10: Sources of DataChapter 11: Dealing with Messy DataChapter 12: Dealing with Large Data
Chapter 1: R GraphicsChapter 2: Basic Graph FunctionsChapter 3: Beyond the Basics – Adjusting Key ParametersChapter 4: Creating Scatter PlotsChapter 5: Creating Line Graphs and Time Series ChartsChapter 6: Creating Bar, Dot, and Pie ChartsChapter 7: Creating HistogramsChapter 8: Box and Whisker PlotsChapter 9: Creating Heat Maps and Contour Plots
Chapter 1: Time Series AnalysisChapter 3: Forecasting VolumeChapter 4: Big Data – Advanced AnalyticsChapter 5: FX DerivativesChapter 6: Interest Rate Derivatives and ModelsChapter 7: Exotic OptionsChapter 8: Optimal HedgingChapter 9: Fundamental Analysis
Chapter 1: Introducing Machine LearningChapter 2: Managing and Understanding DataChapter 3: Lazy Learning – Classification Using Nearest NeighborsChapter 4: Probabilistic Learning – Classification Using Naive BayesChapter 5: Divide and Conquer – Classification Using Decision Trees and RulesChapter 6: Forecasting Numeric Data – Regression MethodsChapter 7: Black Box Methods – Neural Networks and Support Vector MachinesChapter 8: Finding Patterns – Market Basket Analysis Using Association Rules

Content preview from R: Data Analysis and Visualization

Creating line graphs

Line graphs are generally used to look at trends in data over time, so the x variable is usually time expressed as time of day, date, month, year, and so on. In this recipe, we will see how we can quickly plot such data using the same plot() function that was used in the previous recipe to make scatter plots.

Getting ready

First, we need to load the dailysales.csv example data file (you can download this file from the code download section of the book's companion website):

sales<-read.csv("dailysales.csv", header=TRUE)

As the file's name suggests, it contains daily sales data of a product. It has two columns: a date column and a sales column that shows the number of units sold.