What Is AWS Glue? Serverless ETL and Data Catalog

Definition

AWS Glue is a serverless data integration service — it discovers, catalogs, cleans, transforms, and moves data for analytics and machine learning, with no clusters for you to size, patch, or keep running.

Two things are bundled under the name, and it helps to separate them:

• The Glue Data Catalog — a central metadata store that records what tables you have, what columns and types they contain, and where the underlying files live. It is Hive Metastore-compatible, so Athena, Redshift Spectrum, EMR, and Lake Formation all read the same table definitions instead of each keeping its own.
• The Glue ETL engine — managed Apache Spark that runs your transformation jobs. You write PySpark or Scala, or assemble jobs visually, and Glue provisions the workers for the duration of the run and bills by the second.

The practical pitch is that Glue removes the two least interesting parts of a data pipeline: knowing what your data looks like, and running a Spark cluster to process it. You point a crawler at an S3 prefix, it infers the schema and partitions and writes them to the Catalog, and Athena can query the data immediately — no DDL written by hand.

Glue bills at $0.44 per DPU-hour with a one-minute minimum, and the Data Catalog's first million objects and million requests each month are free, which makes it genuinely cheap to use as a catalog even if you never run an ETL job.

AWS Glue Components

Glue provides multiple components that work together:

• Glue Data Catalog — a persistent, centralized metadata store. Each AWS account gets one Catalog per Region. It contains databases, tables, partitions, and connection definitions. The Catalog is the backbone of the AWS analytics ecosystem.
• Crawlers — automated programs that scan data sources (S3, JDBC databases, DynamoDB, etc.), infer schemas and partitions, and register or update tables in the Data Catalog.
• ETL Jobs — serverless Spark jobs that extract data from sources, apply transformations, and load results into targets. You can write jobs in Python (PySpark) or Scala, or use the visual editor.
• Glue Studio — a visual drag-and-drop interface for building, running, and monitoring ETL jobs without writing code.
• DataBrew — a no-code data preparation tool with 250+ built-in transformations for cleaning and normalizing data.
• Streaming ETL — Glue jobs can consume data continuously from Kinesis Data Streams or Amazon MSK (Kafka) with micro-batch processing.
• Data Quality — built-in rules engine (powered by the open-source Deequ library) that validates data during ETL and publishes metrics to CloudWatch.
• Interactive sessions — serverless Spark backends for notebooks, so you can develop a job interactively and pay per second rather than keeping a development endpoint alive.
• Triggers and workflows — schedule jobs, chain them on completion, or fire them on an event, without standing up an external orchestrator.

A note on AWS Glue Elastic Views

Glue Elastic Views was announced as a preview in 2020 and never reached general availability. It no longer appears in the AWS Glue documentation or the console, so if you found a tutorial referencing it, that tutorial is stale. For the problem it was meant to solve — keeping a derived, queryable copy of data in sync across stores — the current answers are zero-ETL integrations (Aurora and DynamoDB into Redshift, for example), materialized views in Redshift, or an ordinary Glue ETL job on a schedule.

A typical workflow: a crawler scans an S3 bucket, creates tables in the Data Catalog, an ETL job transforms the raw data into Parquet and writes it to a curated S3 prefix, and Athena or Redshift Spectrum queries the curated data using the same Catalog tables.

Key Features and Limits

• DynamicFrame — Glue's extension of Spark DataFrame that handles semi-structured and inconsistent schemas gracefully.
• Job bookmarks — track previously processed data to enable incremental ETL without reprocessing entire datasets.
• Workflow orchestration — Glue Workflows chain crawlers and jobs with triggers (scheduled, on-demand, or event-based).
• Connections — JDBC, MongoDB, Kafka, Kinesis, custom connectors from AWS Marketplace.
• Python Shell jobs — lightweight jobs for small-scale transforms or API calls that don't need Spark overhead.
• Auto Scaling — Glue 4.0+ jobs auto-scale workers based on workload, reducing over-provisioning.
• Security — encryption at rest (SSE-S3, SSE-KMS) and in transit, VPC endpoints, Lake Formation fine-grained access control, IAM policies.
• Limits — maximum 1,000 databases per Catalog, 3 million tables per database, 10 million partitions per table. Jobs can run up to 48 hours.

Common Use Cases

1. Data lake ETL — crawl raw data in S3, transform to Parquet/Iceberg, load into curated zones.
2. Data catalog for analytics — central metadata store shared by Athena, Redshift, EMR.
3. Database migration prep — extract from RDS/on-prem databases, transform schemas, load into S3 or Redshift.
4. Streaming data preparation — consume Kinesis or Kafka streams, apply transformations, deliver to S3 or Redshift in near-real-time.
5. No-code data cleaning — DataBrew for analysts who need to cleanse and normalize data without writing code.
6. Data quality monitoring — embed quality checks into ETL pipelines to catch schema drift or data anomalies early.
7. Cross-account data sharing — share Catalog tables via Lake Formation and RAM for multi-account architectures.

AWS Glue Pricing: Is AWS Glue Free?

Partly, and the free part is the useful one. The Data Catalog is free for the first million objects stored and the first million requests per month — a threshold most teams never cross, since an "object" is a table, partition, or database, not a data file. You can therefore use Glue purely as the metastore behind Athena and Redshift Spectrum and pay nothing.

What is not free is compute. Anything that runs Spark bills by the DPU-hour:

| What runs | Rate | Billing granularity | | --- | --- | --- | | ETL jobs | $0.44 per DPU-hour | Per second, 1-minute minimum | | Crawlers | $0.44 per DPU-hour | Per second, 1-minute minimum | | Interactive sessions | $0.44 per DPU-hour | Per second, 1-minute minimum | | Data Catalog storage (over 1M objects) | $1.00 per 100,000 objects per month | Monthly | | Data Catalog statistics & table optimization | $0.44 per DPU-hour | Per second | | DataBrew interactive sessions | $1.00 per 30-minute session | Per session | | DataBrew jobs | $0.48 per node-hour | Per minute |

One DPU is 4 vCPUs and 16 GB of memory. A concrete example: a job using 6 DPUs for 15 minutes costs about $0.66. Streaming ETL uses the same DPU-hour rate but runs continuously, so a single always-on streaming job at 2 DPUs is roughly $0.88/hour — about $640/month — which is the line item that most often surprises people.

The one-minute minimum matters more than it looks. A pipeline of forty tiny jobs that each finish in eight seconds is billed as forty minutes, not five. Batching small transformations into fewer jobs is usually the single biggest Glue cost lever.

Pros and Cons

Pros

• Fully serverless — no clusters to manage, auto-scaling in Glue 4.0+.
• Data Catalog is the de facto standard metastore across AWS analytics services.
• Visual authoring (Glue Studio) and no-code prep (DataBrew) lower the barrier to entry.
• Job bookmarks and workflows provide built-in incremental processing and orchestration.
• Deep integration with Lake Formation for fine-grained security.

Cons

• Spark job cold-start times can be 1-2 minutes even with Glue 4.0 improvements.
• DPU-hour pricing adds up quickly for long-running or always-on streaming jobs.
• Debugging Spark errors in a serverless environment is harder than on a self-managed EMR cluster.
• Crawler schema inference can be imprecise and may require manual corrections.
• Limited control over Spark configuration compared to EMR.

Comparison with Alternatives

| | AWS Glue | Amazon EMR | AWS Step Functions + Lambda | | --- | --- | --- | --- | | Model | Serverless Spark | Managed clusters (EC2/EKS/Serverless) | Serverless orchestration | | Best for | Managed ETL, Data Catalog | Large-scale or custom Hadoop/Spark/Flink | Lightweight orchestration, non-Spark transforms | | Pricing | DPU-hour | EC2 + EMR uplift | Per state transition + Lambda duration | | Flexibility | Medium (Spark + Python Shell) | High (any Hadoop ecosystem tool) | High (any Lambda runtime) | | Startup time | 1-2 min cold start | 5-15 min cluster launch | Milliseconds |

Exam Relevance

• Cloud Practitioner (CLF-C02) — know Glue is a serverless ETL service and that the Data Catalog stores metadata.
• Solutions Architect Associate (SAA-C03) — Glue crawlers populate the Data Catalog, Athena queries use Catalog tables, Glue for S3-to-Redshift ETL pipelines.
• Data Engineer Associate (DEA-C01) — heavy coverage: job bookmarks for incremental ETL, DynamicFrames, Streaming ETL, Data Quality, DataBrew, Glue Studio, Workflows vs Step Functions orchestration.
• Developer Associate (DVA-C02) — Glue Python Shell jobs for lightweight transforms, Catalog API integration.

Common Pitfalls

• Over-provisioned DPUs. Glue bills per DPU-hour, and doubling DPUs rarely halves runtime once a job is I/O-bound. Start small, watch the Spark UI for idle executors, and enable Auto Scaling (Glue 3.0+) instead of guessing a fixed DPU count.
• Job bookmarks left off. Without bookmarks, a job re-reads and re-processes the entire source every run — silently multiplying cost and creating duplicate rows downstream. Enable bookmarks for incremental ETL.
• The small-files problem. Thousands of tiny S3 objects wreck Spark performance and inflate runtime. Compact upstream or use groupFiles/groupSize so each task reads a sensible chunk.
• Crawler cost creep. Crawlers are billed per DPU-hour too. Re-crawling a huge bucket on a tight schedule when the schema never changes is pure waste — narrow the include path or crawl on demand.
• Schema drift breaking jobs. A new column upstream can silently change the Data Catalog schema and break a downstream job. Pin the schema or add explicit ApplyMapping.

A Worked Pricing Example

A Spark ETL job using 10 DPUs that runs for 20 minutes daily (Glue bills per second, 1-minute minimum, ~$0.44/DPU-hour as of 2026, us-east-1):

• 10 DPU × (20 ÷ 60) h = 3.33 DPU-hours × $0.44 = $1.47 per run
• × 30 days ≈ $44/month for the job, plus crawler DPU-hours and Data Catalog storage (first 1 million objects free).

Biggest levers: enable job bookmarks (stop reprocessing), right-size DPUs with Auto Scaling, and consider Glue Flex execution for non-urgent jobs to cut the DPU rate.

Frequently Asked Questions

Q: What is the Glue Data Catalog and why does it matter?

A: The Glue Data Catalog is a centralized metadata repository that stores table definitions, schemas, partition information, and connection details. It matters because it is Hive Metastore-compatible and serves as the shared metastore for Athena, Redshift Spectrum, EMR, and Lake Formation. Instead of each service maintaining its own metadata, they all read from one Catalog, ensuring consistency. The first 1 million objects and 1 million requests per month are free.

Q: How do Glue crawlers work and when should I use them?

A: Crawlers connect to a data source (S3, JDBC, DynamoDB), sample the data, infer its schema (column names, types, partitions), and create or update table definitions in the Data Catalog. Use crawlers when your data sources change frequently or when you want to auto-discover new partitions. However, for well-defined schemas, manually defining tables via DDL or CloudFormation is faster and avoids schema-inference surprises.

Q: Is AWS Glue free?

A: The Data Catalog is free up to the first million objects stored and the first million requests per month, which is enough for most organisations to use Glue as their metastore at no cost. Compute is not free: ETL jobs, crawlers, and interactive sessions all bill at $0.44 per DPU-hour, charged per second with a one-minute minimum, so a 6-DPU job running 15 minutes costs roughly $0.66. There is no perpetual free tier for Glue jobs. The two costs that catch people out are always-on streaming ETL jobs — around $640/month for a modest 2-DPU job — and pipelines made of many very short jobs, where the one-minute minimum dominates the actual runtime.

Q: When should I choose Glue over EMR for ETL?

A: Choose Glue when you want zero infrastructure management, need the Data Catalog, prefer visual authoring (Glue Studio), or run moderate-scale Spark ETL jobs. Choose EMR when you need fine-grained Spark tuning, want to run non-Spark frameworks (Hive, Presto, Flink, HBase), require GPU instances for ML, or when sustained heavy workloads make EMR's EC2-based pricing more cost-effective than Glue's DPU-hour model.

Q: How is AWS Glue priced?

A: Glue bills the Spark/Python-shell ETL jobs and crawlers per DPU-hour (a DPU is 4 vCPU + 16 GB RAM), metered per second with a 1-minute minimum. The Data Catalog is free for the first million objects and first million requests per month. The two biggest cost levers are enabling job bookmarks (so you process only new data) and right-sizing DPUs with Auto Scaling.

Q: What is a DPU in AWS Glue?

A: A DPU (Data Processing Unit) is Glue's unit of compute: 4 vCPUs and 16 GB of memory. A Spark job runs on a cluster of DPUs you request (or that Auto Scaling manages), and you pay per DPU-hour of runtime. More DPUs add parallelism but only speed up a job while it is CPU/memory-bound, not once it is limited by source I/O.

---

This article reflects AWS features and pricing as of 2026. AWS services evolve rapidly — always verify against the official AWS Glue documentation before making production decisions.