book

Python Data Science Handbook

by Jake VanderPlas

November 2016

Beginner to intermediate

548 pages

13h 58m

English

O'Reilly Media, Inc.

Read now

Unlock full access

What Is Data Science?Who Is This Book For?Why Python?Python 2 Versus Python 3Outline of This BookUsing Code ExamplesInstallation ConsiderationsConventions Used in This BookO’Reilly SafariHow to Contact Us
Shell or Notebook?Launching the IPython ShellLaunching the Jupyter NotebookHelp and Documentation in IPythonAccessing Documentation with ?Accessing Source Code with ??Exploring Modules with Tab CompletionKeyboard Shortcuts in the IPython ShellNavigation ShortcutsText Entry ShortcutsCommand History ShortcutsMiscellaneous ShortcutsIPython Magic CommandsPasting Code Blocks: %paste and %cpasteRunning External Code: %runTiming Code Execution: %timeitHelp on Magic Functions: ?, %magic, and %lsmagicInput and Output HistoryIPython’s In and Out ObjectsUnderscore Shortcuts and Previous OutputsSuppressing OutputRelated Magic CommandsIPython and Shell CommandsQuick Introduction to the ShellShell Commands in IPythonPassing Values to and from the ShellShell-Related Magic CommandsErrors and DebuggingControlling Exceptions: %xmodeDebugging: When Reading Tracebacks Is Not EnoughProfiling and Timing CodeTiming Code Snippets: %timeit and %timeProfiling Full Scripts: %prunLine-by-Line Profiling with %lprunProfiling Memory Use: %memit and %mprunMore IPython ResourcesWeb ResourcesBooks
Understanding Data Types in PythonA Python Integer Is More Than Just an IntegerA Python List Is More Than Just a ListFixed-Type Arrays in PythonCreating Arrays from Python ListsCreating Arrays from ScratchNumPy Standard Data TypesThe Basics of NumPy ArraysNumPy Array AttributesArray Indexing: Accessing Single ElementsArray Slicing: Accessing SubarraysReshaping of ArraysArray Concatenation and SplittingComputation on NumPy Arrays: Universal FunctionsThe Slowness of LoopsIntroducing UFuncsExploring NumPy’s UFuncsAdvanced Ufunc FeaturesUfuncs: Learning MoreAggregations: Min, Max, and Everything in BetweenSumming the Values in an ArrayMinimum and MaximumExample: What Is the Average Height of US Presidents?Computation on Arrays: BroadcastingIntroducing BroadcastingRules of BroadcastingBroadcasting in PracticeComparisons, Masks, and Boolean LogicExample: Counting Rainy DaysComparison Operators as ufuncsWorking with Boolean ArraysBoolean Arrays as MasksFancy IndexingExploring Fancy IndexingCombined IndexingExample: Selecting Random PointsModifying Values with Fancy IndexingExample: Binning DataSorting ArraysFast Sorting in NumPy: np.sort and np.argsortPartial Sorts: PartitioningExample: k-Nearest NeighborsStructured Data: NumPy’s Structured ArraysCreating Structured ArraysMore Advanced Compound TypesRecordArrays: Structured Arrays with a TwistOn to Pandas
Installing and Using PandasIntroducing Pandas ObjectsThe Pandas Series ObjectThe Pandas DataFrame ObjectThe Pandas Index ObjectData Indexing and SelectionData Selection in SeriesData Selection in DataFrameOperating on Data in PandasUfuncs: Index PreservationUFuncs: Index AlignmentUfuncs: Operations Between DataFrame and SeriesHandling Missing DataTrade-Offs in Missing Data ConventionsMissing Data in PandasOperating on Null ValuesHierarchical IndexingA Multiply Indexed SeriesMethods of MultiIndex CreationIndexing and Slicing a MultiIndexRearranging Multi-IndicesData Aggregations on Multi-IndicesCombining Datasets: Concat and AppendRecall: Concatenation of NumPy ArraysSimple Concatenation with pd.concatCombining Datasets: Merge and JoinRelational AlgebraCategories of JoinsSpecification of the Merge KeySpecifying Set Arithmetic for JoinsOverlapping Column Names: The suffixes KeywordExample: US States DataAggregation and GroupingPlanets DataSimple Aggregation in PandasGroupBy: Split, Apply, CombinePivot TablesMotivating Pivot TablesPivot Tables by HandPivot Table SyntaxExample: Birthrate DataVectorized String OperationsIntroducing Pandas String OperationsTables of Pandas String MethodsExample: Recipe DatabaseWorking with Time SeriesDates and Times in PythonPandas Time Series: Indexing by TimePandas Time Series Data StructuresFrequencies and OffsetsResampling, Shifting, and WindowingWhere to Learn MoreExample: Visualizing Seattle Bicycle CountsHigh-Performance Pandas: eval() and query()Motivating query() and eval(): Compound Expressionspandas.eval() for Efficient OperationsDataFrame.eval() for Column-Wise OperationsDataFrame.query() MethodPerformance: When to Use These FunctionsFurther Resources
General Matplotlib TipsImporting matplotlibSetting Stylesshow() or No show()? How to Display Your PlotsSaving Figures to FileTwo Interfaces for the Price of OneSimple Line PlotsAdjusting the Plot: Line Colors and StylesAdjusting the Plot: Axes LimitsLabeling PlotsSimple Scatter PlotsScatter Plots with plt.plotScatter Plots with plt.scatterplot Versus scatter: A Note on EfficiencyVisualizing ErrorsBasic ErrorbarsContinuous ErrorsDensity and Contour PlotsVisualizing a Three-Dimensional FunctionHistograms, Binnings, and DensityTwo-Dimensional Histograms and BinningsCustomizing Plot LegendsChoosing Elements for the LegendLegend for Size of PointsMultiple LegendsCustomizing ColorbarsCustomizing ColorbarsExample: Handwritten DigitsMultiple Subplotsplt.axes: Subplots by Handplt.subplot: Simple Grids of Subplotsplt.subplots: The Whole Grid in One Goplt.GridSpec: More Complicated ArrangementsText and AnnotationExample: Effect of Holidays on US BirthsTransforms and Text PositionArrows and AnnotationCustomizing TicksMajor and Minor TicksHiding Ticks or LabelsReducing or Increasing the Number of TicksFancy Tick FormatsSummary of Formatters and LocatorsCustomizing Matplotlib: Configurations and StylesheetsPlot Customization by HandChanging the Defaults: rcParamsStylesheetsThree-Dimensional Plotting in MatplotlibThree-Dimensional Points and LinesThree-Dimensional Contour PlotsWireframes and Surface PlotsSurface TriangulationsGeographic Data with BasemapMap ProjectionsDrawing a Map BackgroundPlotting Data on MapsExample: California CitiesExample: Surface Temperature DataVisualization with SeabornSeaborn Versus MatplotlibExploring Seaborn PlotsExample: Exploring Marathon Finishing TimesFurther ResourcesMatplotlib ResourcesOther Python Graphics Libraries
What Is Machine Learning?Categories of Machine LearningQualitative Examples of Machine Learning ApplicationsSummaryIntroducing Scikit-LearnData Representation in Scikit-LearnScikit-Learn’s Estimator APIApplication: Exploring Handwritten DigitsSummaryHyperparameters and Model ValidationThinking About Model ValidationSelecting the Best ModelLearning CurvesValidation in Practice: Grid SearchSummaryFeature EngineeringCategorical FeaturesText FeaturesImage FeaturesDerived FeaturesImputation of Missing DataFeature PipelinesIn Depth: Naive Bayes ClassificationBayesian ClassificationGaussian Naive BayesMultinomial Naive BayesWhen to Use Naive BayesIn Depth: Linear RegressionSimple Linear RegressionBasis Function RegressionRegularizationExample: Predicting Bicycle TrafficIn-Depth: Support Vector MachinesMotivating Support Vector MachinesSupport Vector Machines: Maximizing the MarginExample: Face RecognitionSupport Vector Machine SummaryIn-Depth: Decision Trees and Random ForestsMotivating Random Forests: Decision TreesEnsembles of Estimators: Random ForestsRandom Forest RegressionExample: Random Forest for Classifying DigitsSummary of Random ForestsIn Depth: Principal Component AnalysisIntroducing Principal Component AnalysisPCA as Noise FilteringExample: EigenfacesPrincipal Component Analysis SummaryIn-Depth: Manifold LearningManifold Learning: “HELLO”Multidimensional Scaling (MDS)MDS as Manifold LearningNonlinear Embeddings: Where MDS FailsNonlinear Manifolds: Locally Linear EmbeddingSome Thoughts on Manifold MethodsExample: Isomap on FacesExample: Visualizing Structure in DigitsIn Depth: k-Means ClusteringIntroducing k-Meansk-Means Algorithm: Expectation–MaximizationExamplesIn Depth: Gaussian Mixture ModelsMotivating GMM: Weaknesses of k-MeansGeneralizing E–M: Gaussian Mixture ModelsGMM as Density EstimationExample: GMM for Generating New DataIn-Depth: Kernel Density EstimationMotivating KDE: HistogramsKernel Density Estimation in PracticeExample: KDE on a SphereExample: Not-So-Naive BayesApplication: A Face Detection PipelineHOG FeaturesHOG in Action: A Simple Face DetectorCaveats and ImprovementsFurther Machine Learning ResourcesMachine Learning in PythonGeneral Machine Learning

Content preview from Python Data Science Handbook

Chapter 4. Visualization with Matplotlib

We’ll now take an in-depth look at the Matplotlib tool for visualization in Python. Matplotlib is a multiplatform data visualization library built on NumPy arrays, and designed to work with the broader SciPy stack. It was conceived by John Hunter in 2002, originally as a patch to IPython for enabling interactive MATLAB-style plotting via gnuplot from the IPython command line. IPython’s creator, Fernando Perez, was at the time scrambling to finish his PhD, and let John know he wouldn’t have time to review the patch for several months. John took this as a cue to set out on his own, and the Matplotlib package was born, with version 0.1 released in 2003. It received an early boost when it was adopted as the plotting package of choice of the Space Telescope Science Institute (the folks behind the Hubble Telescope), which financially supported Matplotlib’s development and greatly expanded its capabilities.

One of Matplotlib’s most important features is its ability to play well with many operating systems and graphics backends. Matplotlib supports dozens of backends and output types, which means you can count on it to work regardless of which operating system you are using or which output format you wish. This cross-platform, everything-to-everyone approach has been one of the great strengths of Matplotlib. It has led to a large userbase, which in turn has led to an active developer base and Matplotlib’s powerful tools and ubiquity within the ...