8
Integrating Continous Integration into Your Workflow
As we grow our projects, many data projects go from being a scattering of notebooks to a continuous integration (CI)-driven application. In this chapter, we will go through some of the tooling and concepts for stringing together your Python scripts and notebooks into a working data application. We will be using Jenkins for CI, GitHub for source control, workflows for orchestration, and Terraform for Infrastructure as Code (IaC). Those tools can be swapped out for your preferred tool without much effort.
In this chapter, we’re going to cover the following main topics:
- Python wheels and creating a Python package
- CI with Jenkins
- Working with source control using GitHub
- Creating Databricks ...
Get Modern Data Architectures with Python now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
