book

Data Wrangling with JavaScript

Name: Data Wrangling with JavaScript
Author: Ashley Davis
ISBN: 9781617294846

by Ashley Davis

December 2018

Beginner to intermediate

432 pages

12h 32m

English

Manning Publications

Read now

Unlock full access

Titlepage
Copyright
preface
acknowledgments
about this book
Who should read this bookHow this book is organized: a roadmapAbout the codeBook forumOther online resources
about the author
about the cover illustration
Chapter 1: Getting started: establishing your data pipeline
1.1 Why data wrangling?1.2 What’s data wrangling?1.3 Why a book on JavaScript data wrangling?1.4 What will you get out of this book?1.5 Why use JavaScript for data wrangling?1.6 Is JavaScript appropriate for data analysis?1.7 Navigating the JavaScript ecosystem1.8 Assembling your toolkit1.9 Establishing your data pipeline1.9.1 Setting the stage1.9.2 The data-wrangling process1.9.3 Planning1.9.4 Acquisition, storage, and retrieval1.9.5 Exploratory coding1.9.6 Clean and prepare1.9.7 Analysis1.9.8 Visualization1.9.9 Getting to productionSummary
Chapter 2: Getting started with Node.js
2.1 Starting your toolkit2.2 Building a simple reporting system2.3 Getting the code and data2.3.1 Viewing the code2.3.2 Downloading the code2.3.3 Installing Node.js2.3.4 Installing dependencies2.3.5 Running Node.js code2.3.6 Running a web application2.3.7 Getting the data2.3.8 Getting the code for chapter 22.4 Installing Node.js2.4.1 Checking your Node.js version2.5 Working with Node.js2.5.1 Creating a Node.js project2.5.2 Creating a command-line application2.5.3 Creating a code library2.5.4 Creating a simple web server2.6 Asynchronous coding2.6.1 Loading a single file2.6.2 Loading multiple files2.6.3 Error handling2.6.4 Asynchronous coding with promises2.6.5 Wrapping asynchronous operations in promises2.6.6 Async coding with “async” and “await”Summary
Chapter 3: Acquisition, storage, and retrieval
3.1 Building out your toolkit3.2 Getting the code and data3.3 The core data representation3.3.1 The earthquakes website3.3.2 Data formats covered3.3.3 Power and flexibility3.4 Importing data3.4.1 Loading data from text files3.4.2 Loading data from a REST API3.4.3 Parsing JSON text data3.4.4 Parsing CSV text data3.4.5 Importing data from databases3.4.6 Importing data from MongoDB3.4.7 Importing data from MySQL3.5 Exporting data3.5.1 You need data to export!3.5.2 Exporting data to text files3.5.3 Exporting data to JSON text files3.5.4 Exporting data to CSV text files3.5.5 Exporting data to a database3.5.6 Exporting data to MongoDB3.5.7 Exporting data to MySQL3.6 Building complete data conversions3.7 Expanding the processSummary

Chapter 4: Working with unusual data
4.1 Getting the code and data4.2 Importing custom data from text files4.3 Importing data by scraping web pages4.3.1 Identifying the data to scrape4.3.2 Scraping with Cheerio4.4 Working with binary data4.4.1 Unpacking a custom binary file4.4.2 Packing a custom binary file4.4.3 Replacing JSON with BSON4.4.4 Converting JSON to BSON4.4.5 Deserializing a BSON fileSummary
Chapter 5: Exploratory coding
5.1 Expanding your toolkit5.2 Analyzing car accidents5.3 Getting the code and data5.4 Iteration and your feedback loop5.5 A first pass at understanding your data5.6 Working with a reduced data sample5.7 Prototyping with Excel5.8 Exploratory coding with Node.js5.8.1 Using Nodemon5.8.2 Exploring your data5.8.3 Using Data-Forge5.8.4 Computing the trend column5.8.5 Outputting a new CSV file5.9 Exploratory coding in the browserPutting it all togetherSummary
Chapter 6: Clean and prepare
6.1 Expanding our toolkit6.2 Preparing the reef data6.3 Getting the code and data6.4 The need for data cleanup and preparation6.5 Where does broken data come from?6.6 How does data cleanup fit into the pipeline?6.7 Identifying bad data6.8 Kinds of problems6.9 Responses to bad dataTechniques for fixing bad dataCleaning our data set6.11.1 Rewriting bad rows6.11.2 Filtering rows of data6.11.3 Filtering columns of dataPreparing our data for effective use6.12.1 Aggregating rows of data6.12.2 Combining data from different files using globby6.12.3 Splitting data into separate filesBuilding a data processing pipeline with Data-ForgeSummary
Chapter 7: Dealing with huge data files
7.1 Expanding our toolkit7.2 Fixing temperature data7.3 Getting the code and data7.4 When conventional data processing breaks down7.5 The limits of Node.js7.5.1 Incremental data processing7.5.2 Incremental core data representation7.5.3 Node.js file streams basics primer7.5.4 Transforming huge CSV files7.5.5 Transforming huge JSON files7.5.6 Mix and matchSummary
Chapter 8: Working with a mountain of data
8.1 Expanding our toolkit8.2 Dealing with a mountain of data8.3 Getting the code and data8.4 Techniques for working with big data8.4.1 Start small8.4.2 Go back to small8.4.3 Use a more efficient representation8.4.4 Prepare your data offline8.5 More Node.js limitations8.6 Divide and conquer8.7 Working with large databases8.7.1 Database setup8.7.2 Opening a connection to the database8.7.3 Moving large files to your database8.7.4 Incremental processing with a database cursor8.7.5 Incremental processing with data windows8.7.6 Creating an index8.7.7 Filtering using queries8.7.8 Discarding data with projection8.7.9 Sorting large data sets8.8 Achieving better data throughput8.8.1 Optimize your code8.8.2 Optimize your algorithm8.8.3 Processing data in parallelSummary
Chapter 9: Practical data analysis
9.1 Expanding your toolkit9.2 Analyzing the weather data9.3 Getting the code and data9.4 Basic data summarization9.4.1 Sum9.4.2 Average9.4.3 Standard deviation9.5 Group and summarize9.6 The frequency distribution of temperatures9.7 Time series9.7.1 Yearly average temperature9.7.2 Rolling average9.7.3 Rolling standard deviation9.7.4 Linear regression9.7.5 Comparing time series9.7.6 Stacking time series operations9.8 Understanding relationships9.8.1 Detecting correlation with a scatter plot9.8.2 Types of correlation9.8.3 Determining the strength of the correlation9.8.4 Computing the correlation coefficientSummary
Chapter 10: Browser-based visualization
10.1 Expanding your toolkit10.2 Getting the code and data10.3 Choosing a chart type10.4 Line chart for New York City temperature10.4.1 The most basic C3 line chart10.4.2 Adding real data10.4.3 Parsing the static CSV file10.4.4 Adding years as the X axis10.4.5 Creating a custom Node.js web server10.4.6 Adding another series to the chart10.4.7 Adding a second Y axis to the chart10.4.8 Rendering a time series chart10.5 Other chart types with C310.5.1 Bar chart10.5.2 Horizontal bar chart10.5.3 Pie chart10.5.4 Stacked bar chart10.5.5 Scatter plot chart10.6 Improving the look of our charts10.7 Moving forward with your own projectsSummary
Chapter 11: Server-side visualization
11.1 Expanding your toolkit11.2 Getting the code and data11.3 The headless browser11.4 Using Nightmare for server-side visualization11.4.1 Why Nightmare?11.4.2 Nightmare and Electron11.4.3 Our process: capturing visualizations with Nightmare11.4.4 Prepare a visualization to render11.4.5 Starting the web server11.4.6 Procedurally start and stop the web server11.4.7 Rendering the web page to an image11.4.8 Before we move on . . .11.4.9 Capturing the full visualizationFeeding the chart with dataMultipage reportsDebugging code in the headless browserMaking it work on a Linux server11.5 You can do much more with a headless browser11.5.1 Web scraping11.5.2 Other usesSummary
Chapter 12: Live data
12.1 We need an early warning system12.2 Getting the code and data12.3 Dealing with live data12.4 Building a system for monitoring air quality12.5 Set up for development12.6 Live-streaming data12.6.1 HTTP POST for infrequent data submission12.6.2 Sockets for high-frequency data submission12.7 Refactor for configuration12.8 Data capture12.9 An event-based architectureCode restructure for event handling12.10.1 Triggering SMS alerts12.10.2 Automatically generating a daily reportLive data processingLive visualizationSummary
Chapter 13: Advanced visualization with D3
13.1 Advanced visualization13.2 Getting the code and data13.3 Visualizing space junk13.4 What is D3?13.5 The D3 data pipeline13.6 Basic setup13.7 SVG crash course13.7.1 SVG circle13.7.2 Styling13.7.3 SVG text13.7.4 SVG group13.8 Building visualizations with D313.8.1 Element state13.8.2 Selecting elements13.8.3 Manually adding elements to our visualization13.8.4 Scaling to fit13.8.5 Procedural generation the D3 way13.8.6 Loading a data file13.8.7 Color-coding the space junk13.8.8 Adding interactivity13.8.9 Adding a year-by-year launch animationSummary
Chapter 14: Getting to production
14.1 Production concerns14.2 Taking our early warning system to production14.3 Deployment14.4 Monitoring14.5 Reliability14.5.1 System longevity14.5.2 Practice defensive programming14.5.3 Data protection14.5.4 Testing and automation14.5.5 Handling unexpected errors14.5.6 Designing for process restart14.5.7 Dealing with an ever-growing database14.6 Security14.6.1 Authentication and authorization14.6.2 Privacy and confidentiality14.6.3 Secret configuration14.7 Scaling14.7.1 Measurement before optimization14.7.2 Vertical scaling14.7.3 Horizontal scalingSummary
appendix a: JavaScript cheat sheet
UpdatesLoggingObjectsArraysRegular expressionsRead and write text files (Node.js, synchronous)Read and write JSON files (Node.js, synchronous)Read and write CSV files (Node.js, synchronous)
appendix b: Data-Forge cheat sheet
UpdatesLoading data into a DataFrameLoading CSV filesLoading JSON filesData transformationData filteringRemoving a columnSaving CSV filesSaving JSON files
appendix c: Getting started with Vagrant
UpdatesInstalling VirtualBoxInstalling VagrantCreating a virtual machineInstalling software on your virtual machineRunning code on your virtual machineTurning off your virtual machine
Index
List of Figures
List of TablesList of Listings

Content preview from Data Wrangling with JavaScript

preface

Data is all around us and growing at an ever-increasing rate. It’s more important than ever before for businesses to deal with data quickly and effectively to understand their customers, monitor their processes, and support decision-making.

If Python and R are the kings of the data world, why, then, should you use JavaScript instead? What role does it play in business, and why do you need to read Data Wrangling with JavaScript?

I’ve used JavaScript myself in various situations. I started with it when I was a game developer building our UIs with web technologies. I soon graduated to Node.js backends to manage collection and processing of metrics and telemetry. We also created analytics dashboards to visualize the data we collected. By ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781617294846Publisher Support Publisher Website

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Data Wrangling with JavaScript

by Ashley Davis

preface

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.