October 2023
Beginner to advanced
3h 36m
English
conference
Data Superstream: The Data Engineering Lifecycle
by Shingai Manjengwa, Joe Reis, Matt Housley, David Mueller, Maria Vechtomova, Dean Bentley-Falcke, Christina Lin, Anitha Sivasubramanian, Sedef Akinli Kocak
Closed Captioning available in German, English, Spanish, French, Japanese, Korean, Portuguese (Portugal, Brazil), Chinese (Simplified), Chinese (Traditional)
Overview
Sponsored by Redpanda
Millions (if not billions) of touch points from customers, systems, and processes enter the average business’s data stream every day. Farther down that stream, analysts, data scientists, and ML engineers take that data and use it to develop hypotheses, identify insights, feed learning models, and so much more. The job of the data engineer is to manage this lifecycle from initial generation through storage to ingestion, transformation, and finally serving the data, using tools like AWS, Azure, Google Cloud, Spark, Kafka, SQL, and many more. It’s extremely important and no small feat. That’s why data engineering is one of the fastest growing jobs—and why data engineers are employed by many of the most recognizable tech companies in the world, including IBM, Amazon, Microsoft, Apple, Google, and Facebook.
Join experienced industry experts to learn how the data engineering lifecycle fits into the overall data lifecycle, explore the technologies you’ll need to conquer along the path from generation to service, and better understand how to meet the needs of analysts, scientists, and ML engineers as well as the business stakeholders and customers driving decisions.
What you’ll learn and how you can apply it- Discover how the data engineering lifecycle allows data professionals to design and build a robust architecture
- Standardize the process of ML model deployment and monitoring with MLOps
- Learn essential data preprocessing techniques crucial for harnessing the potential of LLMs
- You're a data engineer, ML engineer, or data scientist.
- You want to effectively approach the data lifecycle from ingestion to labeling to solving problems with machine learning.
- You want to learn more about prompt engineering and management to tame the inherent unpredictability of AI-generated outputs.
Recommended follow-up:
- Read Fundamentals of Data Engineering (book)
- Read Designing Machine Learning Systems (book)
- Read Machine Learning Design Patterns (book)
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Watch now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.