book

Reproducible Data Science with Pachyderm

by Svetlana Karslioglu

March 2022

Intermediate to advanced

364 pages

6h 44m

English

Packt Publishing

Read now

Unlock full access

ContributorsAbout the authorAbout the reviewers
Who this book is forWhat this book coversTo get the most out of this bookDownload the example code filesDownload the color imagesConventions usedGet in touchShare Your Thoughts
Why is reproducibility important?What is a model?The main principles of reproducibilityThe reproducibility crisis in scienceData fishingBetter reproducibility in science research guidelinesCommon practices to improve reproducibilityDemystifying MLOpsTypes of data science platformsEnd-to-end platformsPluggable solutionsData ingestion toolsData transformation toolsModel serving toolsData monitoring toolsPutting it all togetherExplaining ethical AITrustworthy AISummaryFurther reading
Reviewing Pachyderm architectureWhy can't I use Git for my data pipelines?Pachyderm architecture diagramKubernetesHelmPachyderm internalsOther componentsContainer runtimesLearning about version control primitivesRepositoryBranchCommitDiscovering pipeline elementsTypes of pipelinesDatumSummaryFurther reading
Pipeline specification overview Understanding inputspfsExploring informational parametersnamedescriptionmetadataExploring transformationimagestdinerr_cmderr_stdinenvsecretsimage_pull_secretsaccept_return_codedebuguserworking_dirdockerfileOptimizing your pipelineparallelism_specreprocess_speccache_sizemax_queue_sizechunk_specresource_limitsresource_requestssidecar_resource_limitsscheduling_specjob_timeoutdatum_timeoutdatum_triesExploring service parametersenable_statspod_patch Exploring output parametersSummaryFurther reading
Technical requirementsInstalling the required toolsInstalling Homebrew (macOS only)Installing Windows Subsystem for Linux (for Windows only)Installing the Kubernetes command-line toolInstalling Helm v3Installing minikubeInstalling Docker DesktopInstalling Docker Desktop for macOSInstalling the Pachyderm command-line interfaceEnabling autocompletion for Pachyderm Enabling Pachyderm autocompletion for bashEnabling Pachyderm autocompletion for zshPreparing the Kubernetes environmentEnabling Kubernetes on Docker DesktopEnabling Kubernetes using minikubeDeploying PachydermAccessing the Pachyderm ConsoleDeleting an existing Pachyderm deploymentSummaryFurther reading
Technical requirementsInstalling the required toolsInstalling the AWS Command Line Interface to manage AWSInstalling the AWS IAM authenticator for KubernetesInstalling eksctl to manage Amazon EKSInstalling the Google Cloud SDK to manage Google CloudInstalling the Azure CLI to manage Microsoft AzureDeploying Pachyderm on Amazon EKSPreparing an Amazon EKS cluster to run PachydermCreating an S3 object storage bucket Deploying the clusterDeleting a Pachyderm deployment on Amazon EKSDeploying Pachyderm on GKEPreparing a GKE cluster to run PachydermCreating a Google Cloud object storage bucket Deploying the clusterDeleting a Pachyderm deployment on GKEDeploying Pachyderm on Microsoft AKSPreparing an AKS cluster to run PachydermCreating an Azure storage container Deploying the clusterDeleting a Pachyderm deployment on AKSAccessing the Pachyderm console SummaryFurther reading
Technical requirementsPipeline overviewCreating a repositoryCreating a pipeline specificationViewing the pipeline resultAdding another pipeline stepCleaning upSummaryFurther reading

Technical requirementsDownloading the source filesReviewing the standard Pachyderm workflowExecuting data operationsUploading data to PachydermAbout data lineageExploring data lineageMounting a Pachyderm repository to a local filesystemExecuting pipeline operationsUpdating your pipeline specificationUpdating your codeRunning maintenance operationsTroubleshooting your pipelineUpgrading your Pachyderm clusterCleaning upSummaryFurther reading
Technical requirementsAdjusting virtual machine parametersNLP example overviewIntroduction to NLPLearning the NLP phasesReviewing the NLP exampleCreating repositories and pipelinesCreating the data cleaning pipelineCreating the POS tagging pipelineCreating an NER pipelineRetraining an NER modelCreating the retrain pipelineDeploying the retrained pipelineCleaning upSummaryFurther reading
Technical requirementsReviewing hyperparameter tuning techniques and strategiesGrid searchRandom searchBayesian optimizationRegression evaluation metricsCreating a hyperparameter tuning pipeline in PachydermExample overviewCreating an exploratory analysis pipelineCreating a data cleaning pipelineCreating a pipeline that removes outliersCreating a training pipelineCreating an evaluation pipelineCleaning upSummaryFurther reading
Technical requirementsDownloading the source filesUsing the Pachyderm Go clientInstalling Go on your computerConfiguring $GOPATHCloning the Pachyderm source repositoryConnecting to Pachyderm with the Go client Creating a repository with the Go clientPutting data into a Pachyderm repository with the Go clientCreating pipelines with the Go clientCleaning up the cluster with the Go clientUsing the Pachyderm Python clientInstalling the Pachyderm Python clientConnecting to your Pachyderm cluster with the Python clientCreating a Pachyderm repository with the Python clientPutting data into a Pachyderm repository with the Python clientCreating pipelines with the Pachyderm Python clientCleaning up the cluster with the Python clientSummaryFurther reading
Technical requirementsDownloading the source filesEnabling Pachyderm Notebooks in Pachyderm HubCreate a workspaceConnect to your Pachyderm Hub workspace with pachctlConnect to a Pachyderm notebookRunning basic Pachyderm operations in Pachyderm NotebooksUsing the integrated terminalUsing Pachyderm NotebooksCreating and running an example pipeline in Pachyderm NotebooksPipeline methodologyCreating the pipelinesSummaryFurther reading
Other Books You May EnjoyPackt is searching for authors like youShare Your Thoughts

Content preview from Reproducible Data Science with Pachyderm

Preface

Pachyderm is a distributed version control platform for building end-to-end data science workflows. Since its creation in 2016, Pachyderm has become a go-to solution for large and small organizations. The core functionality of Pachyderm is open source and has a vivid community of engineers around it. This book walks you through basic and advanced examples of Pachyderm usage. This book will help you get started quickly and integrate a reliable data science solution into your infrastructure.

Reproducible Data Science with Pachyderm provides a clear overview of Pachyderm, as well as instructions on how to install and run Pachyderm in the cloud, and how to use the Pachyderm Software-as-a-Service (SaaS) version – Pachyderm Hub. This book ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Start your free trial

Publisher Resources

ISBN: 9781801074483Supplemental Content

Reproducible Data Science with Pachyderm

by Svetlana Karslioglu

Preface

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

You might also like

TensorFlow 2 Pocket Reference

Understanding Economic Equilibrium

Productive and Efficient Data Science with Python: With Modularizing, Memory profiles, and Parallel/GPU Processing

The Esoteric Investor: Alternative Investments for Global Macro Investors

Publisher Resources

Preface

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,and much more.

You might also like

TensorFlow 2 Pocket Reference

Understanding Economic Equilibrium

Productive and Efficient Data Science with Python: With Modularizing, Memory profiles, and Parallel/GPU Processing

The Esoteric Investor: Alternative Investments for Global Macro Investors

Publisher Resources

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.