November 2023
Beginner to intermediate
344 pages
3h 47m
Korean
Content preview from 데이터 엔지니어를 위한 97가지 조언
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
06 파이프라인 테스트를 자동화하라
톰 화이트(Tom White)
데이터 파이프라인을 구축할 때 이번 섹션의 지침을 따라서 데이터 엔지니어링을 소프트웨어 엔지니어링처럼 취급하면, 잘 짜여진 안정적이고 강력한 파이프라인을 작성할 수 있습니다.
¦ 파이프라인 종단 간 테스트를 구축하라 ¦
이 단계에서는 파이프라인이 수행하는 작업 자체에 공을 들이지 마세요. 알려진 입력을 제공하고, 간단한 변환을 수행하고, 출력이 예상과 같은지 테스트하는 등 인프라에 집중하는 게 좋습니다. Junit이나 pytest와 같은 표준적인 단위 테스트 프레임워크를 사용하세요.
¦ 적은 양의 대표 데이터를 사용하라 ¦
대표 데이터의 크기는 테스트를 몇 분만에 실행할 수 있을 만큼 적어야 합니다. 실제 (프로덕션) 시스템에서 가져온 데이터(단, 익명으로 처리된 것이어야 함)라면 가장 좋습니다.
¦ 이진 형식보다 텍스트 형식 데이터를 선호하라 ¦
데이터 파일을 비교할 수 있어야 테스트에 실패했을 때 일어나는 일을 빠르게 파악할 수 있습니다. 입력 및 예상 출력을 확인해서 버전 관리 시스템에 두면 시간이 지나면서 변경되는 내용을 추적할 수 있습니다. 파이프라인에서 이진binary 형식만 받아들이거나 생성한다면 파이프라인 자체에 텍스트 지원을 추가하거나 ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.