November 2023
Beginner to intermediate
344 pages
3h 47m
Korean
Content preview from 데이터 엔지니어를 위한 97가지 조언
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
28 데이터 레이크 아키텍처를 받아들여라
비노트 찬다르(Vinoth Chandar)
종종 데이터 엔지니어는 데이터 파이프라인을 구축해서 외부 소스로부터 데이터를 추출하고, 그 데이터를 변환한 다음, 조직의 다른 부서에서 결과 데이터셋을 쿼리할 수 있게 만듭니다. 단기적으로는 모든 과정을 단일 스테이지 파이프라인으로 구축하는 편이 쉽지만, 이 모델을 수 테라바이트나 페타바이트에 걸친 수천 개의 데이터셋까지 확장하려면 좀 더 신중하게 고민한 데이터 아키텍처가 필요합니다.
¦ 일반적인 함정 ¦
단일 스테이지 접근법을 사용할 때 빠지기 쉬운 일반적인 함정을 알아보겠습니다. 우선, 파이프라인에 입력되는 데이터를 얻으려면 RDBMS나 NoSQL 저장소 등 업스트림 데이터베이스를 스캔해야 하는데, 스캔 과정에서 업스트림 시스템에 압박을 주고 서비스 중단까지 일으킬 수 있기 때문에 단일 스테이지 파이프라인은 확장에 제한이 있습니다. 게다가 직접 데이터에 접근하면 파이프라인 전반에 걸친 표준화(표준 타임스탬프나 키 필드 등)가 이뤄지지 않으며, 스키마 및 데이터 계약contract이 없기 때문에 데이터의 손상 위험이 커집니다. 마지막으로 어떤 통찰을 얻기 위해 자유롭게 데이터에서 상호 연관성을 찾거나 머신 러닝 모델을 설계하기 위해 모든 데이터나 ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.