Effective Data Science Infrastructure

6 Going to production

This chapter covers

• Deploying workflows to a highly scalable and highly available production scheduler
• Setting up a centralized metadata service to track experiments company-wide
• Defining stable execution environments with various software dependencies
• Leveraging versioning to allow multiple people to develop multiple versions of a project safely

Thus far we have been starting all workflows on a personal workstation, maybe a laptop. However, it is not a good idea to run business-critical applications in a prototyping environment. The reasons are many: laptops get lost, they are hard to control and manage centrally, and, more fundamentally, the needs of rapid, human-in-the-loop prototyping are very different from the needs ...