November 2023
Beginner to intermediate
344 pages
3h 47m
Korean
Content preview from 데이터 엔지니어를 위한 97가지 조언
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
07 데이터 파이프라인의 배치 모델을 신중히 검토하라
라고담 머시(Raghotham Murthy)
데이터 레코드를 배치batch로 수집하고 배치 데이터 파이프라인을 구축하는 경우라면 일정 기간에 대한 배치를 생성하는 방법을 정해야 합니다. 배치는 레코드의 data_timestamp나 arrival_timestamp에 기반을 두는데, data_timestamp는 레코드가 자체적으로 갖는 최종 갱신 시점의 타임스탬프이고, arrival_timestamp는 처리 시스템이 레코드를 수신한 시점을 레코드에 붙인 것입니다.
¦ 데이터 시간 윈도우 배치 모델 ¦
데이터 시간 윈도우Data Time Window, DTW 배치 모델에서는 레코드의 data_timestamp가 윈도우 범위에 들어가는 모든 레코드를 수신하면 윈도우에 대한 배치가 생성됩니다. 다음과 같은 경우 이 배치 모델을 사용합니다.
• 데이터를 출처에서 가져오는 경우(출처에서 밀어 보내는 것이 아님)
• 추출 로직에서 data_timestamp가 시간 윈도우 밖에 있는 레코드를 필터링할 수 있음
일례로 데이터베이스에서 윈도우 내의 모든 트랜잭션을 추출할 때 DTW 배치를 사용합니다. DTW 배치는 주어진 시간 윈도우에 속하는 레코드가 모두 해당 배치에 존재한다고 보장되기 ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.