book

Python for Data Analysis, 2nd Edition

by Wes McKinney

October 2017

Beginner to intermediate

547 pages

12h 16m

English

O'Reilly Media, Inc.

Read now

Unlock full access

New for the Second EditionConventions Used in This BookUsing Code ExamplesO’Reilly SafariHow to Contact UsAcknowledgmentsIn Memoriam: John D. Hunter (1968–2012)Acknowledgments for the Second Edition (2017)Acknowledgments for the First Edition (2012)
1.1 What Is This Book About?What Kinds of Data?1.2 Why Python for Data Analysis?Python as GlueSolving the “Two-Language” ProblemWhy Not Python?1.3 Essential Python LibrariesNumPypandasmatplotlibIPython and JupyterSciPyscikit-learnstatsmodels1.4 Installation and SetupWindowsApple (OS X, macOS)GNU/LinuxInstalling or Updating Python PackagesPython 2 and Python 3Integrated Development Environments (IDEs) and Text Editors1.5 Community and Conferences1.6 Navigating This BookCode ExamplesData for ExamplesImport ConventionsJargon
2.1 The Python Interpreter2.2 IPython BasicsRunning the IPython ShellRunning the Jupyter NotebookTab CompletionIntrospectionThe %run CommandExecuting Code from the ClipboardTerminal Keyboard ShortcutsAbout Magic CommandsMatplotlib Integration2.3 Python Language BasicsLanguage SemanticsScalar TypesControl Flow
3.1 Data Structures and SequencesTupleListBuilt-in Sequence FunctionsdictsetList, Set, and Dict Comprehensions3.2 FunctionsNamespaces, Scope, and Local FunctionsReturning Multiple ValuesFunctions Are ObjectsAnonymous (Lambda) FunctionsCurrying: Partial Argument ApplicationGeneratorsErrors and Exception Handling3.3 Files and the Operating SystemBytes and Unicode with Files3.4 Conclusion
4.1 The NumPy ndarray: A Multidimensional Array ObjectCreating ndarraysData Types for ndarraysArithmetic with NumPy ArraysBasic Indexing and SlicingBoolean IndexingFancy IndexingTransposing Arrays and Swapping Axes4.2 Universal Functions: Fast Element-Wise Array Functions4.3 Array-Oriented Programming with ArraysExpressing Conditional Logic as Array OperationsMathematical and Statistical MethodsMethods for Boolean ArraysSortingUnique and Other Set Logic4.4 File Input and Output with Arrays4.5 Linear Algebra4.6 Pseudorandom Number Generation4.7 Example: Random WalksSimulating Many Random Walks at Once4.8 Conclusion
5.1 Introduction to pandas Data StructuresSeriesDataFrameIndex Objects5.2 Essential FunctionalityReindexingDropping Entries from an AxisIndexing, Selection, and FilteringInteger IndexesArithmetic and Data AlignmentFunction Application and MappingSorting and RankingAxis Indexes with Duplicate Labels5.3 Summarizing and Computing Descriptive StatisticsCorrelation and CovarianceUnique Values, Value Counts, and Membership5.4 Conclusion
6.1 Reading and Writing Data in Text FormatReading Text Files in PiecesWriting Data to Text FormatWorking with Delimited FormatsJSON DataXML and HTML: Web Scraping6.2 Binary Data FormatsUsing HDF5 FormatReading Microsoft Excel Files6.3 Interacting with Web APIs6.4 Interacting with Databases6.5 Conclusion
7.1 Handling Missing DataFiltering Out Missing DataFilling In Missing Data7.2 Data TransformationRemoving DuplicatesTransforming Data Using a Function or MappingReplacing ValuesRenaming Axis IndexesDiscretization and BinningDetecting and Filtering OutliersPermutation and Random SamplingComputing Indicator/Dummy Variables7.3 String ManipulationString Object MethodsRegular ExpressionsVectorized String Functions in pandas7.4 Conclusion
8.1 Hierarchical IndexingReordering and Sorting LevelsSummary Statistics by LevelIndexing with a DataFrame’s columns8.2 Combining and Merging DatasetsDatabase-Style DataFrame JoinsMerging on IndexConcatenating Along an AxisCombining Data with Overlap8.3 Reshaping and PivotingReshaping with Hierarchical IndexingPivoting “Long” to “Wide” FormatPivoting “Wide” to “Long” Format8.4 Conclusion
9.1 A Brief matplotlib API PrimerFigures and SubplotsColors, Markers, and Line StylesTicks, Labels, and LegendsAnnotations and Drawing on a SubplotSaving Plots to Filematplotlib Configuration9.2 Plotting with pandas and seabornLine PlotsBar PlotsHistograms and Density PlotsScatter or Point PlotsFacet Grids and Categorical Data9.3 Other Python Visualization Tools9.4 Conclusion

10.1 GroupBy MechanicsIterating Over GroupsSelecting a Column or Subset of ColumnsGrouping with Dicts and SeriesGrouping with FunctionsGrouping by Index Levels10.2 Data AggregationColumn-Wise and Multiple Function ApplicationReturning Aggregated Data Without Row Indexes10.3 Apply: General split-apply-combineSuppressing the Group KeysQuantile and Bucket AnalysisExample: Filling Missing Values with Group-Specific ValuesExample: Random Sampling and PermutationExample: Group Weighted Average and CorrelationExample: Group-Wise Linear Regression10.4 Pivot Tables and Cross-TabulationCross-Tabulations: Crosstab10.5 Conclusion
11.1 Date and Time Data Types and ToolsConverting Between String and Datetime11.2 Time Series BasicsIndexing, Selection, SubsettingTime Series with Duplicate Indices11.3 Date Ranges, Frequencies, and ShiftingGenerating Date RangesFrequencies and Date OffsetsShifting (Leading and Lagging) Data11.4 Time Zone HandlingTime Zone Localization and ConversionOperations with Time Zone−Aware Timestamp ObjectsOperations Between Different Time Zones11.5 Periods and Period ArithmeticPeriod Frequency ConversionQuarterly Period FrequenciesConverting Timestamps to Periods (and Back)Creating a PeriodIndex from Arrays11.6 Resampling and Frequency ConversionDownsamplingUpsampling and InterpolationResampling with Periods11.7 Moving Window FunctionsExponentially Weighted FunctionsBinary Moving Window FunctionsUser-Defined Moving Window Functions11.8 Conclusion
12.1 Categorical DataBackground and MotivationCategorical Type in pandasComputations with CategoricalsCategorical Methods12.2 Advanced GroupBy UseGroup Transforms and “Unwrapped” GroupBysGrouped Time Resampling12.3 Techniques for Method ChainingThe pipe Method12.4 Conclusion
13.1 Interfacing Between pandas and Model Code13.2 Creating Model Descriptions with PatsyData Transformations in Patsy FormulasCategorical Data and Patsy13.3 Introduction to statsmodelsEstimating Linear ModelsEstimating Time Series Processes13.4 Introduction to scikit-learn13.5 Continuing Your Education
14.1 1.USA.gov Data from BitlyCounting Time Zones in Pure PythonCounting Time Zones with pandas14.2 MovieLens 1M DatasetMeasuring Rating Disagreement14.3 US Baby Names 1880–2010Analyzing Naming Trends14.4 USDA Food Database14.5 2012 Federal Election Commission DatabaseDonation Statistics by Occupation and EmployerBucketing Donation AmountsDonation Statistics by State14.6 Conclusion
A.1 ndarray Object InternalsNumPy dtype HierarchyA.2 Advanced Array ManipulationReshaping ArraysC Versus Fortran OrderConcatenating and Splitting ArraysRepeating Elements: tile and repeatFancy Indexing Equivalents: take and putA.3 BroadcastingBroadcasting Over Other AxesSetting Array Values by BroadcastingA.4 Advanced ufunc Usageufunc Instance MethodsWriting New ufuncs in PythonA.5 Structured and Record ArraysNested dtypes and Multidimensional FieldsWhy Use Structured Arrays?A.6 More About SortingIndirect Sorts: argsort and lexsortAlternative Sort AlgorithmsPartially Sorting Arraysnumpy.searchsorted: Finding Elements in a Sorted ArrayA.7 Writing Fast NumPy Functions with NumbaCreating Custom numpy.ufunc Objects with NumbaA.8 Advanced Array Input and OutputMemory-Mapped FilesHDF5 and Other Array Storage OptionsA.9 Performance TipsThe Importance of Contiguous Memory
B.1 Using the Command HistorySearching and Reusing the Command HistoryInput and Output VariablesB.2 Interacting with the Operating SystemShell Commands and AliasesDirectory Bookmark SystemB.3 Software Development ToolsInteractive DebuggerTiming Code: %time and %timeitBasic Profiling: %prun and %run -pProfiling a Function Line by LineB.4 Tips for Productive Code Development Using IPythonReloading Module DependenciesCode Design TipsB.5 Advanced IPython FeaturesMaking Your Own Classes IPython-FriendlyProfiles and ConfigurationB.6 Conclusion

Content preview from Python for Data Analysis, 2nd Edition

Preface

Section 1. New for the Second Edition

The first edition of this book was published in 2012, during a time when open source data analysis libraries for Python (such as pandas) were very new and developing rapidly. In this updated and expanded second edition, I have overhauled the chapters to account both for incompatible changes and deprecations as well as new features that have occurred in the last five years. I’ve also added fresh content to introduce tools that either did not exist in 2012 or had not matured enough to make the first cut. Finally, I have tried to avoid writing about new or cutting-edge open source projects that may not have had a chance to mature. I would like readers of this edition to find that the content is still almost as relevant in 2020 or 2021 as it is in 2017.

The major updates in this second edition include:

All code, including the Python tutorial, updated for Python 3.6 (the first edition used Python 2.7)
Updated Python installation instructions for the Anaconda Python Distribution and other needed Python packages
Updates for the latest versions of the pandas library in 2017
A new chapter on some more advanced pandas tools, and some other usage tips
A brief introduction to using statsmodels and scikit-learn

I also reorganized a significant portion of the content from the first edition to make the book more accessible to newcomers.

Section 2. Conventions Used in This Book

The following typographical conventions are used in this book:

Italic: Indicates new ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Start your free trial

Publisher Resources