March 2025
Intermediate to advanced
408 pages
10h 29m
Portuguese (Portugal, Brazil)
Content preview from Guia de estudo do Databricks Certified Data Engineer Associate
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
Capítulo 5. Processamento de dados incrementais
Este trabalho foi traduzido com recurso a IA. Agradecemos o teu feedback e comentários: translation-feedback@oreilly.com
Nos capítulos anteriores, explorámos os fundamentos do processamento de dados em grupos ou lotes de uma só vez. No entanto, quando os dados são gerados continuamente, as abordagens tradicionais de processamento em lote tendem a se tornar insuficientes. Neste capítulo, exploraremos os conceitos e as técnicas de processamento de dados de fluxo contínuo, incluindo o fluxo contínuo estruturado do Spark e a ingestão incremental de dados de arquivos. Além disso, discutiremos o conceito de arquitetura medalhão e como construí-la sob o modelo de processamento de fluxo.
Streaming de dados com o Apache Spark
O Apache Spark fornece um suporte robusto para o processamento de dados de fluxo contínuo, permitindo-te realizar análises em tempo real de forma eficiente. No centro desse processo está o conceito de um fluxo de dados, que é o foco do processamento. Para trabalhar efetivamente com dados de streaming no Spark, vamos primeiro entender o que é um fluxo de dados e suas caraterísticas.
O que é um fluxo de dados?
Um fluxo de dados representa um fluxo ilimitado de dados, muitas vezes proveniente de várias fontes, como sensores, ficheiros de registo ou plataformas de redes sociais. À medida que novos dados são gerados, eles são anexados ao fluxo, tornando-o um conjunto de dados dinâmico e em constante mudança. Exemplos de ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.