It’s 2025, and the “good enough” era of data engineering is over. If you are still running your production pipelines on AWS Glue 4.0 , you are not just using older software; you are likely overpaying for performance that no longer meets modern standards.

AWS Glue 5.0 has fundamentally changed the ROI of ETL (Extract, Transform, Load) jobs by integrating the latest compute engines and smarter data management features. That includes Apache Spark 3.5, Python 3.11, Java 17, and more upgrades.

Today, the focus is on extreme cost-efficiency and deep integration with transactional data lakes. If your team is still hesitant about the switch, this guide breaks down why the upgrade to Glue 5.0 is no longer optional for high-velocity data teams.

Tech Stack Upgrades: Apache Spark 3.5 and Beyond

The immediate benefit of Glue 5.0 is the foundational upgrade to its data processing engine. While Glue 4.0 relied on Spark 3.3, Glue 5.0 jumps to Apache Spark 3.5.*, which brings a suite of optimizations for query planning and execution. The result: your AWS Glue 5.0 jobs will run faster and cost lower in comparison to Glue 4.0.

FeatureAWS Glue 4.0AWS Glue 5.0
Spark VersionSpark 3.3.0Spark 3.5.4
Python VersionPython 3.10Python 3.11
Java RuntimeJava 8 / 11Java 17
Default OSAmazon Linux 2Amazon Linux 2023

The transition to Java 17 is particularly impactful as well. It offers much more efficient memory management and garbage collection compared to Java 8. That means your heavy shuffling jobs are less likely to hit the dreaded OutOfMemory (OOM) errors that plagued jobs running on older AWS Glue versions with older Java versions.

Furthermore, Python 3.11 introduces significant speed improvements for native Python code, making your UDFs (User Defined Functions) run noticeably faster.

Performance Benchmarks: Faster Execution Saves Money

In the world of AWS Glue, time is quite literally money. Because Glue is billed by DPU-Hour (Data Processing Unit per hour), any reduction in jobs’ run time directly lowers your monthly bill. AWS Glue 5.0 offers much improved performance such as:

  • Higher Throughput: Early benchmarks show that the Spark 3.5 engine in Glue 5.0 provides roughly a 30% performance boost for standard ETL workloads.
  • Flex Execution Efficiency: Glue 5.0 refines the “Flex” execution class, allowing you to run non-urgent jobs on spare capacity for a fraction of the cost.
  • Better Worker Sizing: The new engine handles “G.1X” and “G.2X” workers more effectively, reducing the dead time spent on resource provisioning.

If a job that took 60 minutes on Glue 4.0 now takes 42 minutes on Glue 5.0, you are saving 18 minutes of DPU billing every single time that job runs. For enterprise-scale pipelines, that can add up to thousands of dollars in monthly savings.

New Features: Record-level Indexing and Iceberg Branching

Beyond raw speed, Glue 5.0 introduces features that solve the unseen headaches of data engineering such as semantic lookups, “small files” problem and complex version control. It offers new maintenance features like automatic compaction and optimized data writes, thanks to upgraded runtimes of Apache Iceberg and Delta Lake.

Record-level Indexing

This feature allows Glue to perform semantic lookups across your data lake much faster than traditional full-table scans. It’s a lifesaver for scenarios where you need to update or delete specific records within a massive S3-based data lake. One of the scenarios is jobs performing incremental data refreshes like in Delta Lake.

Iceberg Branching

In recent years, we are seeing a massive shift toward transactional formats like Apache Iceberg and Delta Lake. Glue 5.0 provides native support for Iceberg branching. This means you can create a development branch of your data lake, run your ETL tests, and merge them back into the main branch once verified – all within AWS Glue.

Tips for your Migration Strategy

Migrating from Glue 4.0 to 5.0 is relatively straightforward, but there are a few gotchas you need to be aware of before flipping the switch in production.

  1. Check for Deprecations: Some older libraries and specific ML transforms (like the old FindMatches pattern) have been deprecated or moved. Review the official AWS migration guide to ensure your custom scripts do not break.
  2. Update Docker Images: If you develop or test locally, you must switch to the Glue 5.0 Docker image. This ensures that the environment you test in matches the Amazon Linux 2023 environment used in the newer AWS Glue.
  3. Test your Connectors: While most JDBC connectors are compatible, the move to Java 17 may require you to update specific third-party .jar files.

Pro Tip: Start by migrating your smallest, lowest-risk jobs first. Monitor the logs for ClassCastExceptions error which are common when upgrading Java.

Upgrading to Glue 5.0 is not just about staying current; it’s about giving your team the best possible tools to handle the data demands of 2025. The decision is clear, thanks to the cost savings of Spark 3.5 and the benefits of record-level indexing. After all, he longer you wait, the more technical debt (and AWS costs) you accumulate.

References

  1. Introducing AWS Glue 5.0 for Apache Spark [ AWS Big Data Blog (original) (archived) ]

  2. Migrating AWS Glue for Spark jobs to AWS Glue version 5.0 [ AWS Glue User Guide (original) (archived) ]

Let’s discuss.

Get in touch to discuss an idea or project. We can work together to make it live! Or enquire about writing guest posts or speaking in meetups or workshops.