book

Data Science at the Command Line

by Jeroen Janssens

October 2014

Beginner to intermediate

210 pages

4h 32m

English

O'Reilly Media, Inc.

Read now

Unlock full access

What to Expect from This BookHow to Read This BookWho This Book Is ForConventions Used in This BookUsing Code ExamplesSafari® Books OnlineHow to Contact UsAcknowledgments
OverviewData Science Is OSEMNObtaining DataScrubbing DataExploring DataModeling DataInterpreting DataIntermezzo ChaptersWhat Is the Command Line?Why Data Science at the Command Line?The Command Line Is AgileThe Command Line Is AugmentingThe Command Line Is ScalableThe Command Line Is ExtensibleThe Command Line Is UbiquitousA Real-World Use CaseFurther Reading
OverviewSetting Up Your Data Science ToolboxStep 1: Download and Install VirtualBoxStep 2: Download and Install VagrantStep 3: Download and Start the Data Science ToolboxStep 4: Log In (on Linux and Mac OS X)Step 4: Log In (on Microsoft Windows)Step 5: Shut Down or Start AnewEssential Concepts and ToolsThe EnvironmentExecuting a Command-Line ToolFive Types of Command-Line ToolsCombining Command-Line ToolsRedirecting Input and OutputWorking with FilesHelp!Further Reading
OverviewCopying Local Files to the Data Science ToolboxLocal Version of Data Science ToolboxRemote Version of Data Science ToolboxDecompressing FilesConverting Microsoft Excel SpreadsheetsQuerying Relational DatabasesDownloading from the InternetCalling Web APIsFurther Reading
OverviewConverting One-Liners into Shell ScriptsStep 1: Copy and PasteStep 2: Add Permission to ExecuteStep 3: Define ShebangStep 4: Remove Fixed InputStep 5: ParameterizeStep 6: Extend Your PATHCreating Command-Line Tools with Python and RPorting the Shell ScriptProcessing Streaming Data from Standard InputFurther Reading
OverviewCommon Scrub Operations for Plain TextFiltering LinesExtracting ValuesReplacing and Deleting ValuesWorking with CSVBodies and Headers and Columns, Oh My!Performing SQL Queries on CSVWorking with HTML/XML and JSONCommon Scrub Operations for CSVExtracting and Reordering ColumnsFiltering LinesMerging ColumnsCombining Multiple CSV FilesFurther Reading
OverviewIntroducing DrakeInstalling DrakeObtain Top Ebooks from Project GutenbergEvery Workflow Starts with a Single StepWell, That DependsRebuilding Specific TargetsDiscussionFurther Reading
OverviewInspecting Data and Its PropertiesHeader or Not, Here I ComeInspect All the DataFeature Names and Data TypesUnique Identifiers, Continuous Variables, and FactorsComputing Descriptive StatisticsUsing csvstatUsing R from the Command Line with RioCreating VisualizationsIntroducing Gnuplot and feedgnuplotIntroducing ggplot2HistogramsBar PlotsDensity PlotsBox PlotsScatter PlotsLine GraphsSummaryFurther Reading
OverviewSerial ProcessingLooping Over NumbersLooping Over LinesLooping Over FilesParallel ProcessingIntroducing GNU ParallelSpecifying InputControlling the Number of Concurrent JobsLogging and OutputCreating Parallel ToolsDistributed ProcessingGet a List of Running AWS EC2 InstancesRunning Commands on Remote MachinesDistributing Local Data Among Remote MachinesProcessing Files on Remote MachinesDiscussionFurther Reading
OverviewMore Wine, Please!Dimensionality Reduction with TapkeeIntroducing TapkeeInstalling TapkeeLinear and Nonlinear MappingsClustering with WekaIntroducing WekaTaming Weka on the Command LineConverting Between CSV and ARFFComparing Three Clustering AlgorithmsRegression with SciKit-Learn LaboratoryPreparing the DataRunning the ExperimentParsing the ResultsClassification with BigMLCreating Balanced Train and Test Data SetsCalling the APIInspecting the ResultsConclusionFurther Reading

Let’s RecapThree Pieces of AdviceBe PatientBe CreativeBe PracticalWhere to Go from Here?APIsShell ProgrammingPython, R, and SQLInterpreting DataGetting in Touch
aliasawkawsbashbcbigmlerbodycatcdchmodcolscowsaycpcsvcutcsvgrepcsvjoincsvlookcsvsortcsvsqlcsvstackcsvstatcurlcurlicuecutdisplaydrakedseqechoenvexportfeedgnuplotfieldsplitfindforgitgrepheadheaderin2csvjqjson2csvlesslsmanmkdirmvparallelpastepbcpippwdpythonRRioRio-scatterrmrun_experimentsamplescpscrapesedseqshufsortsplitsql2csvsshsudotailtapkeetarteetrtreetypeuniqunpackunrarunzipwcwekawhichxml2json

Content preview from Data Science at the Command Line

Chapter 3. Obtaining Data

This chapter deals with the first step of the OSEMN model: obtaining data. After all, without any data, there is not much data science that we can do. We assume that the data that is needed to solve the data science problem at hand already exists at some location in some form. Our goal is to get this data onto your computer (or into your Data Science Toolbox) in a form that we can work with.

According to the Unix philosophy, text is a universal interface. Almost every command-line tool takes text as input, produces text as output, or both. This is the main reason why command-line tools can work so well together. However, as we’ll see, even just text can come in multiple forms.

Data can be obtained in several ways—for example by downloading it from a server, by querying a database, or by connecting to a web API. Sometimes, the data comes in a compressed form or in a binary format such as Microsoft Excel. In this chapter, we discuss several tools that help tackle this from the command line, including: curl (Stenberg, 2012), in2csv (Groskopf, 2014), sql2csv (Groskopf, 2014), and tar (Bailey, Eggert, & Poznyakoff, 2014).