The Evolving Role of the Data Engineer

Appendix. Best Practices for Managing Resources

This report has laid out a dizzying platter of tools available for data engineering, but until now has avoided the question of what computer systems to run the tools on. Like any computing operation, you need physical resources in order to do ingestion and transformations. Nearly everyone now schedules resources through either containers or virtual machines (VMs), on-premises or in the cloud, because they allow easy scaling and the efficient exploitation of physical resources.

Containers and Virtual Machines

A container essentially runs a single application in an isolated environment, whereas a VM runs a whole operating system, hosting any applications you want to include. Proponents of VMs claim they are more secure than containers, although attacks against both have been recorded. Containers are more lightweight and can spin up faster.

Platform as a Service (PaaS) is another convenient cloud solution, providing an API on which you can run your programming functions. PaaS makes resource management particularly easy for the programmer, because the vendor handles all of the CPU and memory resources behind the scenes.

All the modern big data tools described in this report have interfaces to popular tools for containers and VMs, both on-premises and among cloud vendors. You need to hook up the tool of your choice to the system of your choice, and specify how much CPU and memory you need for each instance of the job you are running. ...