1 What is data engineering?

This chapter covers

  • What is data engineering?

  • What do data engineers do?

  • How does Microsoft define data engineering?

  • What tools does Azure provide for data engineering?

Data collection is on the rise. More and more systems are generating more and more data every day.1

More than 30,000 gigabytes of data are generated every second, and the rate of data creation is only accelerating.

--Nathan Marz

Increased connectivity has led to increased sophistication and user interaction in software systems. New deployments of connected “smart” electronics also rely on increased connectivity. In response, businesses now collect and store data from all aspects of their products. This has led to an enormous increase in ...

Get Azure Storage, Streaming, and Batch Analytics now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.