book

Data Science at the Command Line

by Jeroen Janssens

October 2014

Beginner to intermediate

210 pages

4h 32m

English

O'Reilly Media, Inc.

Read now

Unlock full access

What to Expect from This BookHow to Read This BookWho This Book Is ForConventions Used in This BookUsing Code ExamplesSafari® Books OnlineHow to Contact UsAcknowledgments
OverviewData Science Is OSEMNObtaining DataScrubbing DataExploring DataModeling DataInterpreting DataIntermezzo ChaptersWhat Is the Command Line?Why Data Science at the Command Line?The Command Line Is AgileThe Command Line Is AugmentingThe Command Line Is ScalableThe Command Line Is ExtensibleThe Command Line Is UbiquitousA Real-World Use CaseFurther Reading
OverviewSetting Up Your Data Science ToolboxStep 1: Download and Install VirtualBoxStep 2: Download and Install VagrantStep 3: Download and Start the Data Science ToolboxStep 4: Log In (on Linux and Mac OS X)Step 4: Log In (on Microsoft Windows)Step 5: Shut Down or Start AnewEssential Concepts and ToolsThe EnvironmentExecuting a Command-Line ToolFive Types of Command-Line ToolsCombining Command-Line ToolsRedirecting Input and OutputWorking with FilesHelp!Further Reading
OverviewCopying Local Files to the Data Science ToolboxLocal Version of Data Science ToolboxRemote Version of Data Science ToolboxDecompressing FilesConverting Microsoft Excel SpreadsheetsQuerying Relational DatabasesDownloading from the InternetCalling Web APIsFurther Reading
OverviewConverting One-Liners into Shell ScriptsStep 1: Copy and PasteStep 2: Add Permission to ExecuteStep 3: Define ShebangStep 4: Remove Fixed InputStep 5: ParameterizeStep 6: Extend Your PATHCreating Command-Line Tools with Python and RPorting the Shell ScriptProcessing Streaming Data from Standard InputFurther Reading
OverviewCommon Scrub Operations for Plain TextFiltering LinesExtracting ValuesReplacing and Deleting ValuesWorking with CSVBodies and Headers and Columns, Oh My!Performing SQL Queries on CSVWorking with HTML/XML and JSONCommon Scrub Operations for CSVExtracting and Reordering ColumnsFiltering LinesMerging ColumnsCombining Multiple CSV FilesFurther Reading
OverviewIntroducing DrakeInstalling DrakeObtain Top Ebooks from Project GutenbergEvery Workflow Starts with a Single StepWell, That DependsRebuilding Specific TargetsDiscussionFurther Reading
OverviewInspecting Data and Its PropertiesHeader or Not, Here I ComeInspect All the DataFeature Names and Data TypesUnique Identifiers, Continuous Variables, and FactorsComputing Descriptive StatisticsUsing csvstatUsing R from the Command Line with RioCreating VisualizationsIntroducing Gnuplot and feedgnuplotIntroducing ggplot2HistogramsBar PlotsDensity PlotsBox PlotsScatter PlotsLine GraphsSummaryFurther Reading
OverviewSerial ProcessingLooping Over NumbersLooping Over LinesLooping Over FilesParallel ProcessingIntroducing GNU ParallelSpecifying InputControlling the Number of Concurrent JobsLogging and OutputCreating Parallel ToolsDistributed ProcessingGet a List of Running AWS EC2 InstancesRunning Commands on Remote MachinesDistributing Local Data Among Remote MachinesProcessing Files on Remote MachinesDiscussionFurther Reading
OverviewMore Wine, Please!Dimensionality Reduction with TapkeeIntroducing TapkeeInstalling TapkeeLinear and Nonlinear MappingsClustering with WekaIntroducing WekaTaming Weka on the Command LineConverting Between CSV and ARFFComparing Three Clustering AlgorithmsRegression with SciKit-Learn LaboratoryPreparing the DataRunning the ExperimentParsing the ResultsClassification with BigMLCreating Balanced Train and Test Data SetsCalling the APIInspecting the ResultsConclusionFurther Reading

Let’s RecapThree Pieces of AdviceBe PatientBe CreativeBe PracticalWhere to Go from Here?APIsShell ProgrammingPython, R, and SQLInterpreting DataGetting in Touch
aliasawkawsbashbcbigmlerbodycatcdchmodcolscowsaycpcsvcutcsvgrepcsvjoincsvlookcsvsortcsvsqlcsvstackcsvstatcurlcurlicuecutdisplaydrakedseqechoenvexportfeedgnuplotfieldsplitfindforgitgrepheadheaderin2csvjqjson2csvlesslsmanmkdirmvparallelpastepbcpippwdpythonRRioRio-scatterrmrun_experimentsamplescpscrapesedseqshufsortsplitsql2csvsshsudotailtapkeetarteetrtreetypeuniqunpackunrarunzipwcwekawhichxml2json

Content preview from Data Science at the Command Line

Chapter 2. Getting Started

In this chapter, we are going to make sure that you have all the prerequisites for doing data science at the command line. The prerequisites fall into two parts: (1) having a proper environment with all the command-line tools that we employ in this book, and (2) understanding the essential concepts that come into play when using the command line.

First, we describe how to install the Data Science Toolbox, which is a virtual environment based on GNU/Linux that contains all the necessary command-line tools. Subsequently, we explain the essential command-line concepts through examples.

By the end of this chapter, you’ll have everything you need in order to continue with the first step of doing data science, namely obtaining data.

Overview

In this chapter, you’ll learn:

How to set up the Data Science Toolbox
Essential concepts and tools necessary to do data science at the command line

Setting Up Your Data Science Toolbox

In this book we use many different command-line tools. The distribution of GNU/Linux that we are using, Ubuntu, comes with a whole bunch of command-line tools pre-installed. Moreover, Ubuntu offers many packages that contain other, relevant command-line tools. Installing these packages yourself is not too difficult. However, we also use command-line tools that are not available as packages and require a more manual, and more involved, installation. In order to acquire the necessary command-line tools without having to go through ...