November 2023
Beginner to intermediate
344 pages
3h 47m
Korean
Content preview from 데이터 엔지니어를 위한 97가지 조언
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
74 데이터 계보의 중요성
질리앵 르 뎀(Julien Le Dem)
데이터 엔지니어는 어떻게 보면 다양한 소스에서 유입되는 데이터셋 수집가와도 같습니다. 문제는 데이터셋이 벽면 보드에 고정되어 있지 않기 때문에 수집이 어렵다는 것입니다. 데이터셋은 때에 맞춰 유지 보수하고 업데이트해야 합니다. 또한, 다양한 사용 사례에 맞춰 변환하고 조정해야 합니다. 데이터셋은 시간이 지남에 따라 형태가 바뀌므로, 그 위에 구축된 모든 계층을 그에 맞춰 업데이트해야 합니다.
데이터 파이프라인이 쌓일수록 굉장히 복잡해지기 때문에 적시에 안정적으로 업데이트되는 상태를 유지하기 어려워집니다. 수집부터 머신 러닝, 비즈니스 인텔리전스, 전반적인 데이터 처리에 이르는 모든 변환 계층에서 데이터 계보lineage를 관측하면 가시성에 대한 중요한 자료를 얻을 수 있습니다. 이 정보를 통해 대기 중인 당직 엔지니어는 위기 상황을 파악하고 신속하게 문제를 해결할 수 있습니다.
데이터셋이 다른 데이터셋에서 어떻게 파생되었는지는 계보를 통해서 이해할 수 있습니다. 운영 중인 계보는 여기서 한 걸음 더 나아가 그 데이터 변환이 언제 어떻게 일어났는지를 추적합니다. 이를 통해 다음과 같은 정보를 얻습니다.
• 사용된 입력 버전
• 읽어 들인 데이터 부분 집합
• 변환을 수행한 ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.