book

Julia for Data Analysis

by Bogumil Kaminski

January 2023

Beginner to intermediate

472 pages

13h 21m

English

Manning Publications

Read now

Unlock full access

forewordprefaceacknowledgmentsabout this bookWho should read this bookHow this book is organized: A roadmapAbout the codeliveBook discussion forumOther online resourcesabout the authorabout the cover illustration
1.1 What is Julia and why is it useful?1.2 Key features of Julia from a data scientist’s perspective1.2.1 Julia is fast because it is a compiled language1.2.2 Julia provides full support for interactive workflows1.2.3 Julia programs are highly reusable and easy to compose together1.2.4 Julia has a built-in state-of-the-art package manager1.2.5 It is easy to integrate existing code with Julia1.3 Usage scenarios of tools presented in the book1.4 Julia’s drawbacks1.5 What data analysis skills will you learn?1.6 How can Julia be used for data analysis?Summary
2.1 Representing values2.2 Defining variables2.3 Using the most important control-flow constructs2.3.1 Computations depending on a Boolean condition2.3.2 Loops2.3.3 Compound expressions2.3.4 A first approach to calculating the winsorized mean2.4 Defining functions2.4.1 Defining functions using the function keyword2.4.2 Positional and keyword arguments of functions2.4.3 Rules for passing arguments to functions2.4.4 Short syntax for defining simple functions2.4.5 Anonymous functions2.4.6 Do blocks2.4.7 Function-naming convention in Julia2.4.8 A simplified definition of a function computing the winsorized mean2.5 Understanding variable scoping rulesSummary
3.1 Understanding Julia’s type system3.1.1 A single function in Julia may have multiple methods3.1.2 Types in Julia are arranged in a hierarchy3.1.3 Finding all supertypes of a type3.1.4 Finding all subtypes of a type3.1.5 Union of types3.1.6 Deciding what type restrictions to put in method signature3.2 Using multiple dispatch in Julia3.2.1 Rules for defining methods of a function3.2.2 Method ambiguity problem3.2.3 Improved implementation of winsorized mean3.3 Working with packages and modules3.3.1 What is a module in Julia?3.3.2 How can packages be used in Julia?3.3.3 Using StatsBase.jl to compute the winsorized mean3.4 Using macrosSummary
4.1 Working with arrays4.1.1 Getting the data into a matrix4.1.2 Computing basic statistics of the data stored in a matrix4.1.3 Indexing into arrays4.1.4 Performance considerations of copying vs. making a view4.1.5 Calculating correlations between variables4.1.6 Fitting a linear regression4.1.7 Plotting the Anscombe’s quartet data4.2 Mapping key-value pairs with dictionaries4.3 Structuring your data by using named tuples4.3.1 Defining named tuples and accessing their contents4.3.2 Analyzing Anscombe’s quartet data stored in a named tuple4.3.3 Understanding composite types and mutability of values in JuliaSummary

5.1 Vectorizing your code using broadcasting5.1.1 Understanding syntax and meaning of broadcasting in Julia5.1.2 Expanding length-1 dimensions in broadcasting5.1.3 Protecting collections from being broadcasted over5.1.4 Analyzing Anscombe’s quartet data using broadcasting5.2 Defining methods with parametric types5.2.1 Most collection types in Julia are parametric5.2.2 Rules for subtyping of parametric types5.2.3 Using subtyping rules to define the covariance function5.3 Integrating with Python5.3.1 Preparing data for dimensionality reduction using t-SNE5.3.2 Calling Python from Julia5.3.3 Visualizing the results of the t-SNE algorithmSummary
6.1 Getting and inspecting the data6.1.1 Downloading files from the web6.1.2 Using common techniques of string construction6.1.3 Reading the contents of a file6.2 Splitting strings6.3 Using regular expressions to work with strings6.3.1 Working with regular expressions6.3.2 Writing a parser of a single line of movies.dat file6.4 Extracting a subset from a string with indexing6.4.1 UTF-8 encoding of strings in Julia6.4.2 Character vs. byte indexing of strings6.4.3 ASCII strings6.4.4 The Char type6.5 Analyzing genre frequency in movies.dat6.5.1 Finding common movie genres6.5.2 Understanding genre popularity evolution over the years6.6 Introducing symbols6.6.1 Creating symbols6.6.2 Using symbols6.7 Using fixed-width string types to improve performance6.7.1 Available fixed-width strings6.7.2 Performance of fixed-width strings6.8 Compressing vectors of strings with PooledArrays.jl6.8.1 Creating a file containing flower names6.8.2 Reading in the data to a vector and compressing it6.8.3 Understanding the internal design of PooledArray6.9 Choosing appropriate storage for collections of stringsSummary
7.1 Understanding the NBP Web API7.1.1 Getting the data via a web browser7.1.2 Getting the data by using Julia7.1.3 Handling cases when an NBP Web API query fails7.2 Working with missing data in Julia7.2.1 Definition of the missing value7.2.2 Working with missing values7.3 Getting time-series data from the NBP Web API7.3.1 Working with dates7.3.2 Fetching data from the NBP Web API for a range of dates7.4 Analyzing data fetched from the NBP Web API7.4.1 Computing summary statistics7.4.2 Finding which days of the week have the most missing values7.4.3 Plotting the PLN/USD exchange rateSummary
8.1 Fetching, unpacking, and inspecting the data8.1.1 Downloading the file from the web8.1.2 Working with bzip2 archives8.1.3 Inspecting the CSV file8.2 Loading the data to a data frame8.2.1 Reading a CSV file into a data frame8.2.2 Inspecting the contents of a data frame8.2.3 Saving a data frame to a CSV file8.3 Getting a column out of a data frame8.3.1 Understanding the data frame’s storage model8.3.2 Treating a data frame column as a property8.3.3 Getting a column by using data frame indexing8.3.4 Visualizing data stored in columns of a data frame8.4 Reading and writing data frames using different formats8.4.1 Apache Arrow8.4.2 SQLiteSummary
9.1 Advanced data frame indexing9.1.1 Getting a reduced puzzles data frame9.1.2 Overview of allowed column selectors9.1.3 Overview of allowed row-subsetting values9.1.4 Making views of data frame objects9.2 Analyzing the relationship between puzzle difficulty and popularity9.2.1 Calculating mean puzzle popularity by its rating9.2.2 Fitting LOESS regressionSummary
10.1 Reviewing the most important ways to create a data frame10.1.1 Creating a data frame from a matrix10.1.2 Creating a data frame from vectors10.1.3 Creating a data frame using a Tables.jl interface10.1.4 Plotting a correlation matrix of data stored in a data frame10.2 Creating data frames incrementally10.2.1 Vertically concatenating data frames10.2.2 Appending a table to a data frame10.2.3 Adding a new row to an existing data frame10.2.4 Storing simulation results in a data frameSummary
11.1 Converting a data frame to other value types11.1.1 Conversion to a matrix11.1.2 Conversion to a named tuple of vectors11.1.3 Other common conversions11.2 Grouping data frame objects11.2.1 Preparing the source data frame11.2.2 Grouping a data frame11.2.3 Getting group keys of a grouped data frame11.2.4 Indexing a grouped data frame with a single value11.2.5 Comparing performance of indexing methods11.2.6 Indexing a grouped data frame with multiple values11.2.7 Iterating a grouped data frameSummary
12.1 Getting and loading the GitHub developers data set12.1.1 Understanding graphs12.1.2 Fetching GitHub developer data from the web12.1.3 Implementing a function that extracts data from a ZIP file12.1.4 Reading the GitHub developer data into a data frame12.2 Computing additional node features12.2.1 Creating a SimpleGraph object12.2.2 Computing features of nodes by using the Graphs.jl package12.2.3 Counting a node’s web and machine learning neighbors12.3 Using the split-apply-combine approach to predict the developer’s type12.3.1 Computing summary statistics of web and machine learning developer features12.3.2 Visualizing the relationship between the number of web and machine learning neighbors of a node12.3.3 Fitting a logistic regression model predicting developer type12.4 Reviewing data frame mutation operations12.4.1 Performing low-level API operations12.4.2 Using the insertcols! function to mutate a data frameSummary
13.1 Getting and preprocessing the police stop data set13.1.1 Loading all required packages13.1.2 Introducing the @chain macro13.1.3 Getting the police stop data set13.1.4 Comparing functions that perform operations on columns13.1.5 Using short forms of operation specification syntax13.2 Investigating the violation column13.2.1 Finding the most frequent violations13.2.2 Vectorizing functions by using the ByRow wrapper13.2.3 Flattening data frames13.2.4 Using convenience syntax to get the number of rows of a data frame13.2.5 Sorting data frames13.2.6 Using advanced functionalities of DataFramesMeta.jl13.3 Preparing data for making predictions13.3.1 Performing initial transformation of the data13.3.2 Working with categorical data13.3.3 Joining data frames13.3.4 Reshaping data frames13.3.5 Dropping rows of a data frame that hold missing values13.4 Building a predictive model of arrest probability13.4.1 Splitting the data into train and test data sets13.4.2 Fitting a logistic regression model13.4.3 Evaluating the quality of a model’s predictions13.5 Reviewing functionalities provided by DataFrames.jlSummary
14.1 Pricing financial options by using a Monte Carlo simulation14.1.1 Calculating the payoff of an Asian option definition14.1.2 Computing the value of an Asian option14.1.3 Understanding GBM14.1.4 Using a numerical approach to computing the Asian option value14.2 Implementing the option pricing simulator14.2.1 Starting Julia with multiple-thread support14.2.2 Computing the option payoff for a single sample of stock prices14.2.3 Computing the option value14.3 Creating a web service serving the Asian option valuation14.3.1 A general approach to building a web service14.3.2 Creating a web service using Genie.jl14.3.3 Running the web service14.4 Using the Asian option pricing web service14.4.1 Sending a single request to the web service14.4.2 Collecting responses to multiple requests from a web service in a data frame14.4.3 Unnesting a column of a data frame14.4.4 Plotting the results of Asian option pricingSummary
A.1 Installing and setting up JuliaA.2 Getting help in and about JuliaA.3 Managing packages in JuliaA.3.1 Project environmentsA.3.2 Activating project environmentsA.3.3 Potential issues with installing packagesA.3.4 Managing packagesA.3.5 Setting up integration with PythonA.3.6 Setting up integration with RA.4 Reviewing standard ways to work with JuliaA.4.1 Using a terminalA.4.2 Using Visual Studio CodeA.4.3 Using Jupyter NotebookA.4.4 Using Pluto notebooks
C.1 Plotting ecosystems in JuliaC.2 Scaling computing with JuliaC.3 Working with databases and data storage formatsC.4 Using data science methodsSummary

Content preview from Julia for Data Analysis

11 Converting and grouping data frames

This chapter covers

Converting data frames to other Julia types
Writing type-stable code
Understanding type piracy
Grouping data frame objects
Working with grouped data frames

In chapter 10, we reviewed various ways that DataFrame objects can be constructed from different data sources. In this chapter, we discuss the reverse process and show how you can create other objects from a data frame (recall from chapter 9 that a data frame can be a DataFrame or its view, that is, a SubDataFrame). You might want to perform such an operation in two scenarios.

In the first scenario, you need to perform analytical tasks provided by functions that do not accept a data frame as input, but instead accept another type, ...