Table of Contents
In recent years, the discussion around data engineering has shifted from simply “moving data” to “securing and governing data at scale.” If your production pipelines are powered by AWS Glue 5.0, you are running on a reliable foundation, but you are missing out on the secure-by-default architecture offered by AWS Glue 5.1.
Since its general availability, AWS Glue 5.1 has redefined how write operations and open table formats (OTFs) are handled. If your team is debating this “minor” version bump, this guide breaks down why the transition to Glue 5.1 is a strategic benefit for high-velocity data teams for running production-ready secure data pipelines.
The Engine Room: Spark 3.5.6
The most immediate technical benefit of Glue 5.1 is the upgrade of its foundational compute engine. While Glue 5.0 established the Spark 3.5 baseline, Glue 5.1 moves the needle to Apache Spark 3.5.6, bringing a suite of features and upgrades like:
- Stability at Scale: Spark 3.5.6 includes critical maintenance patches that resolve memory overhead issues and improve query planning for complex shuffles.
- Faster UDFs: Python 3.11 runtime reduces the “Python tax,” making custom transformations noticeably faster for User Defined Functions (UDFs).
- Runtime Consistency: The environment continues to use Amazon Linux 2023, ensuring your shell scripts and OS-level dependencies work as same.
The Big Breakthrough: Lake Formation Write Support
In previous iterations, AWS Lake Formation provided robust Fine-Grained Access Control (FGAC) for read operations. However, managing write operations (DML and DDL) often required a fragmented mix of IAM policies and S3 bucket-level permissions.
AWS Glue 5.1 solves this by extending FGAC to write operations, thus providing:
- Centralized Governance: You can now manage exactly who can
INSERT,UPDATE, orDELETErecords at the row and column level directly using Lake Formation. - Unified DML/DDL Control: This support covers data manipulation (DML) and schema changes (DDL) using Spark DataFrames and Spark SQL.
- Reduced IAM Complexity: By offloading write-permissions to Lake Formation, your IAM roles become leaner and easier to audit for security compliance.
Modern OTFs: Iceberg Format V3
The adoption of Open Table Formats (OTF) is the standard for data lakes in the modern era. AWS Glue is built to support this shift with Glue 5.1 bringing native integrations for the latest versions of Apache Iceberg, Apache Hudi, and Delta Lake.
The support for Iceberg Format V3 is particularly noteworthy. It features default column values for more flexible schema evolution without breaking downstream consumers and row-lineage tracking that simplifies the audit trail for transactional data.
It also supports geospatial analytics by introducing geometry and geography native data types as first-class citizens for advanced location intelligence. Also, Deletion Vectors (DVs) replaces delete files with compact, highly efficient binary bitmaps that speeds up merge-on-read operations and reduces file layout fragmentation.
| Feature | Version in Glue 5.0 | Version in Glue 5.1 | Benefit and Impact |
|---|---|---|---|
| Apache Iceberg | Version 1.7.1 | Version 1.10.0 | Includes useful features as listed above from Iceberg Format V3 as well as Iceberg Materialized Views. |
| Apache Hudi | Version 0.15.0 | Version 1.0.2 | Enhances Full-Table Access (FTA) and metadata indexing for massive datasets. |
| Delta Lake | Version 3.3.0 | Version 3.3.2 | Provides deeper integration with the latest Spark engine features, Liquid Clustering and more. |
Comparison: Glue 5.0 vs. Glue 5.1
A side-by-side comparison between Glue 5.0 and 5.1 for your quick reference:
| Component | AWS Glue 5.0 | AWS Glue 5.1 |
|---|---|---|
| Spark Version | Spark 3.5.4 | Spark 3.5.6 |
| Python Version | Python 3.11 | Python 3.11 |
| S3 Connector | S3 (Legacy) | S3A (Default) |
| Write FGAC | Not Supported | Supported (Lake Formation) |
Migration Playbook for Glue 5.1
The migration from Glue 5.0 to 5.1 is designed to be a “drop-in” replacement for most workloads. However, there are minimum two specific factors you should account for during your migration to AWS Glue 5.1. Let’s check them in detail below:
Default S3 Connector
Glue 5.1 now utilizes the optimised S3A connector by default (s3a://). This brings enhanced performance and aligns with the latest Amazon EMR optimizations. While this will help accelerate workloads, you should re-verify any custom performance-tuning flags in your Spark configuration that were specific to the older S3 connector.
Migration Checklist
- Toggle the Version: Update the
GlueVersionparameter in your Job Details or CloudFormation templates to5.1. - Test with Docker: Leverage the official Glue 5.1 Docker image to validate your scripts locally before deploying to production.
- Audit IAM Permissions: Ensure your IAM roles have the necessary permissions for new
StartJobRunparameters and Lake Formation write-support.
Pro Tip: Start by migrating your Iceberg or Delta Lake jobs first to get the most benefits. The performance improvements from the updated OTF libraries often provide the most immediate ROI by lowering DPU hours (and thus costs).
If you stay with Glue 5.0 for your production pipelines, it is a safe choice for static, low-risk pipelines. However, if you are building a modern, transactional data lake, AWS Glue 5.1 is the new baseline. With the Spark 3.5.6 engine refinements, the critical Lake Formation write-governance and more, AWS Glue 5.1 provides the security and performance required for the next generation of data integration projects.
References
- Introducing AWS Glue 5.1 [ What’s New with AWS (original) (archived) ]
- AWS Glue versions [ AWS Documentation (original) (archived) ]
↫ Previous post


