Serverless ETL and Analytics with AWS Glue

Chapter 7: Metadata Management

Just as with relational databases, AWS Glue relies on the concepts of databases and tables to organize and manage datasets. That said, these concepts are quite different in their execution. In a relational database, the data to be stored and its descriptors (such as the schema and comments, also known as metadata) are stored and managed together: there is no way to store data without describing it first, and there is no way to add metadata to already written data.

In big data environments, the storage and metadata layers are decoupled. There is no centralized storage system because of dataset size limitations, and data is typically dumped without a format onto distributed, large-scale systems such as Apache Hadoop ...