book

Reproducible Data Science with Pachyderm

by Svetlana Karslioglu

March 2022

Intermediate to advanced

364 pages

6h 44m

English

Packt Publishing

Read now

Unlock full access

ContributorsAbout the authorAbout the reviewers
Who this book is forWhat this book coversTo get the most out of this bookDownload the example code filesDownload the color imagesConventions usedGet in touchShare Your Thoughts
Why is reproducibility important?What is a model?The main principles of reproducibilityThe reproducibility crisis in scienceData fishingBetter reproducibility in science research guidelinesCommon practices to improve reproducibilityDemystifying MLOpsTypes of data science platformsEnd-to-end platformsPluggable solutionsData ingestion toolsData transformation toolsModel serving toolsData monitoring toolsPutting it all togetherExplaining ethical AITrustworthy AISummaryFurther reading
Reviewing Pachyderm architectureWhy can't I use Git for my data pipelines?Pachyderm architecture diagramKubernetesHelmPachyderm internalsOther componentsContainer runtimesLearning about version control primitivesRepositoryBranchCommitDiscovering pipeline elementsTypes of pipelinesDatumSummaryFurther reading
Pipeline specification overview Understanding inputspfsExploring informational parametersnamedescriptionmetadataExploring transformationimagestdinerr_cmderr_stdinenvsecretsimage_pull_secretsaccept_return_codedebuguserworking_dirdockerfileOptimizing your pipelineparallelism_specreprocess_speccache_sizemax_queue_sizechunk_specresource_limitsresource_requestssidecar_resource_limitsscheduling_specjob_timeoutdatum_timeoutdatum_triesExploring service parametersenable_statspod_patch Exploring output parametersSummaryFurther reading
Technical requirementsInstalling the required toolsInstalling Homebrew (macOS only)Installing Windows Subsystem for Linux (for Windows only)Installing the Kubernetes command-line toolInstalling Helm v3Installing minikubeInstalling Docker DesktopInstalling Docker Desktop for macOSInstalling the Pachyderm command-line interfaceEnabling autocompletion for Pachyderm Enabling Pachyderm autocompletion for bashEnabling Pachyderm autocompletion for zshPreparing the Kubernetes environmentEnabling Kubernetes on Docker DesktopEnabling Kubernetes using minikubeDeploying PachydermAccessing the Pachyderm ConsoleDeleting an existing Pachyderm deploymentSummaryFurther reading
Technical requirementsInstalling the required toolsInstalling the AWS Command Line Interface to manage AWSInstalling the AWS IAM authenticator for KubernetesInstalling eksctl to manage Amazon EKSInstalling the Google Cloud SDK to manage Google CloudInstalling the Azure CLI to manage Microsoft AzureDeploying Pachyderm on Amazon EKSPreparing an Amazon EKS cluster to run PachydermCreating an S3 object storage bucket Deploying the clusterDeleting a Pachyderm deployment on Amazon EKSDeploying Pachyderm on GKEPreparing a GKE cluster to run PachydermCreating a Google Cloud object storage bucket Deploying the clusterDeleting a Pachyderm deployment on GKEDeploying Pachyderm on Microsoft AKSPreparing an AKS cluster to run PachydermCreating an Azure storage container Deploying the clusterDeleting a Pachyderm deployment on AKSAccessing the Pachyderm console SummaryFurther reading
Technical requirementsPipeline overviewCreating a repositoryCreating a pipeline specificationViewing the pipeline resultAdding another pipeline stepCleaning upSummaryFurther reading

Technical requirementsDownloading the source filesReviewing the standard Pachyderm workflowExecuting data operationsUploading data to PachydermAbout data lineageExploring data lineageMounting a Pachyderm repository to a local filesystemExecuting pipeline operationsUpdating your pipeline specificationUpdating your codeRunning maintenance operationsTroubleshooting your pipelineUpgrading your Pachyderm clusterCleaning upSummaryFurther reading
Technical requirementsAdjusting virtual machine parametersNLP example overviewIntroduction to NLPLearning the NLP phasesReviewing the NLP exampleCreating repositories and pipelinesCreating the data cleaning pipelineCreating the POS tagging pipelineCreating an NER pipelineRetraining an NER modelCreating the retrain pipelineDeploying the retrained pipelineCleaning upSummaryFurther reading
Technical requirementsReviewing hyperparameter tuning techniques and strategiesGrid searchRandom searchBayesian optimizationRegression evaluation metricsCreating a hyperparameter tuning pipeline in PachydermExample overviewCreating an exploratory analysis pipelineCreating a data cleaning pipelineCreating a pipeline that removes outliersCreating a training pipelineCreating an evaluation pipelineCleaning upSummaryFurther reading
Technical requirementsDownloading the source filesUsing the Pachyderm Go clientInstalling Go on your computerConfiguring $GOPATHCloning the Pachyderm source repositoryConnecting to Pachyderm with the Go client Creating a repository with the Go clientPutting data into a Pachyderm repository with the Go clientCreating pipelines with the Go clientCleaning up the cluster with the Go clientUsing the Pachyderm Python clientInstalling the Pachyderm Python clientConnecting to your Pachyderm cluster with the Python clientCreating a Pachyderm repository with the Python clientPutting data into a Pachyderm repository with the Python clientCreating pipelines with the Pachyderm Python clientCleaning up the cluster with the Python clientSummaryFurther reading
Technical requirementsDownloading the source filesEnabling Pachyderm Notebooks in Pachyderm HubCreate a workspaceConnect to your Pachyderm Hub workspace with pachctlConnect to a Pachyderm notebookRunning basic Pachyderm operations in Pachyderm NotebooksUsing the integrated terminalUsing Pachyderm NotebooksCreating and running an example pipeline in Pachyderm NotebooksPipeline methodologyCreating the pipelinesSummaryFurther reading
Other Books You May EnjoyPackt is searching for authors like youShare Your Thoughts

Content preview from Reproducible Data Science with Pachyderm

Section 1: Introduction to Pachyderm and Reproducible Data Science

This section introduces the basics of Pachyderm, as well as describing the importance of data reproducibility for an enterprise-level data science platform. You will learn what the main pillars of the Pachyderm solution are, including repositories, datums, jobs, and the most important of them all – the pipeline. The chapter also briefly talks about the ethics of AI in terms of reproducibility.

This section comprises the following chapters: