AWS Glue materialized views without the aggregate pipeline

Quick summary.

AWS Glue materialized views are managed Apache Iceberg tables in the AWS Glue Data Catalog that store a SQL query’s pre-computed result and refresh on a catalog-run schedule. Spark engines on Glue, EMR, and Athena Spark rewrite incoming queries to read the view instead of the base tables. Athena SQL does not rewrite, so a BI query must name the view. Either way the aggregation leaves your BI layer. This article covers the syntax, the rewrite check, what our team saw on a subscription analytics platform, and the limits that rule some workloads out. AWS reports up to 8x query acceleration with reduced compute cost.

If your Power BI or QuickSight dashboards take minutes to refresh, your Athena bill climbs every time an analyst opens a report. Meanwhile, the same GROUP BY runs against the same S3 partitions dozens of times a day. As a result, AWS Glue materialized views can help reduce repeated aggregation work in your BI layer.

Our team found those symptoms on a subscription analytics platform. Specifically, tens of millions of rows of campaign data sat in Iceberg tables on Amazon S3, while a dozen-plus Power BI reports queried them through Athena. As a result, every scheduled refresh re-scanned the same fact tables. Engagement details here are generalized.

The conventional fix is an aggregate pipeline: a Glue job writing summary tables, an orchestration schedule, backfill logic, freshness monitoring, and a rewrite of every report. That is weeks of work and permanent maintenance. AWS Glue materialized views, announced 30 November 2025 just ahead of re:Invent, remove most of it.

What are AWS Glue materialized views?

An AWS Glue materialized view is a managed table in the AWS Glue Data Catalog that stores the pre-computed result of a SQL query as an Apache Iceberg table and keeps it current through catalog-managed refreshes.

Two properties separate materialized views in AWS Glue from a summary table you build yourself. The first is that they refresh themselves: you declare an interval in the view definition, and the Glue Data Catalog detects source changes and refreshes on managed compute. No DAG, no trigger, no Lambda.

The second is that Spark finds it on its own. Spark on Glue, EMR, and Athena Spark rewrites an incoming query to read the materialized view whenever the view can satisfy it, so the job still asks for the base table and the engine quietly serves the pre-computed answer. Athena SQL does not rewrite; it reads the view only when the query names it.

How are Glue materialized views different from Glue Data Catalog views?

A Data Catalog view stores a SQL definition and re-runs it on every query. A materialized view stores the result and serves it.

AWS Glue views come in two forms, and teams running Glue Data Catalog views often assume AWS simply renamed the same feature.

A Data Catalog view is a saved query. A materialized view is a saved answer.

How do you create an AWS Glue materialized view?

Use standard Spark SQL from an AWS Glue 5.1 job, a Glue Studio notebook, or an EMR 7.12 cluster, placing the schedule clause before the SELECT.

Athena reads the result as an ordinary Iceberg table but cannot create, refresh, alter, or drop a view, so the creation path runs through Glue or EMR Spark.

Why this works.SCHEDULE REFRESH hands refresh ownership to the Glue Data Catalog, which assumes the definer IAM role recorded in the view definition and runs the refresh for you. The result lands as an Iceberg table in an S3 general purpose bucket or an S3 Tables bucket, inheriting Iceberg snapshot isolation, so readers never see a half-written refresh. Schedules accept hours, days, or weeks, and the minimum automatic interval is one hour.

How to refresh materialized view data on demand

The two refresh commands behave very differently, and the second is destructive. Run it in a non-production database first.

When to use which. Use incremental refresh on append-heavy fact tables where late-arriving data is rare, because it is dramatically cheaper. The catalog decides per run whether incremental is applicable and falls back to a full refresh without telling you, so read lastRefreshType from SHOW TBLPROPERTIES instead of assuming the cheap path ran. Use full refresh after a backfill, a schema change, or any correction that rewrites historical partitions, since incremental refresh tracks changes forward from the last snapshot and will not catch a rewritten past. A full refresh overwrites the whole table and makes previous snapshots unavailable, so it is not a free safety net.

How does automatic query rewrite work in Spark?

Query rewrite lets Spark jobs use the view with no code change, and it is off by default.

Enable it in your session configuration, with the incremental refresh optimizer:

Why this works.The optimizer compares the logical plan of the incoming query against the definitions of available materialized views. When a view’s stored result is a superset of what the query needs, meaning the same or coarser grain and the same or wider filter range, it substitutes the scan target in the physical plan and leaves your submitted SQL untouched.

How to verify it worked. Run EXPLAIN EXTENDED and look for the view name inside the BatchScan operation. If you still see the base table, the rewrite did not fire: check that your query grain is no finer than the view’s, and remember the Spark metadata cache takes around 30 seconds to populate after a view is created.

Slow, expensive data lake dashboards are rarely one bad query. ScriptsHub Technologies tunes AWS analytics platforms across Athena, Glue, and Iceberg, plus the BI layers on top. If your reports recompute the same aggregations every cycle, we can help. Talk to us about AWS data lake consulting.

What changed on the reporting platform after the switch?

Three scheduled Glue aggregation jobs were retired and replaced with two materialized views, with no change to the semantic model or a single DAX measure.

Our team built a daily campaign rollup and a monthly channel summary at the grain the reports expected. What mattered was how little moved above the storage layer: the semantic model and every measure survived untouched.

The rest followed structurally. Three Glue jobs and their schedules were decommissioned, removing a recurring source of 3 a.m. failure alerts. A query answered from a rollup scans a fraction of the fact table’s rows, so the saving comes from arithmetic rather than tuning. DAX and model-level tuning chases the same win inside the semantic model; this reaches it at storage. Quantified figures from this engagement are not published.

<!– EDITORIAL GATE – NEVER RENDERS, STRIPPED AT BUILD [[FILL-VERIFY]] confirm whether the Power BI datasets were repointed at the materialized view, or whether Athena served them without a source change. The Athena SQL engine does not perform automatic query rewrite. [[FILL-METRICS]] before/after refresh durations, Athena bytes-scanned deltas, and monthly cost change from CloudWatch and Cost Explorer. [[FILL-ENGAGEMENT]] client sign-off on the anonymized figures. [[FILL-AUTHORBIO]] named author with a title matching the article’s authority (principal or lead data engineer), a two-line bio, a headshot, and a LinkedIn URL. An unbylined piece making IAM, Iceberg and semantic-layer recommendations is an E-E-A-T liability. [[FILL-WARSTORY]] two or three un-generatable specifics from the build: what broke first, the verbatim error text, and how long the first refresh took over the 40M-row fact table. Do NOT invent these. Resolve by editing the outcome section, then delete this block. –>

How do you monitor and manage a materialized view?

AWS Glue Data Catalog materialized views publish every refresh to CloudWatch, and DESCRIBE EXTENDED reports the definition, refresh status, and last refresh timestamp.

Retiring your orchestration means retiring your alerting, so wire up the managed equivalents. Refresh metrics land in the AWS/Glue namespace under the MaterializedViewRefresh task type, per-run logs go to /aws-glue/materialized-views/<task_run_id>, and EventBridge emits four event types.

Why this works. The catalog treats each refresh as a tracked task run, so start-materialized-view-refresh-task-run, get-materialized-view-refresh-task-run, and list-materialized-view-refresh-task-runs in the AWS CLI give you the same start, status, duration, and error detail a Glue job would have surfaced. Refresh states are RUNNING, SUCCEEDED, FAILED, and STOPPED. Each run also reports dpuSeconds and processedBytes, which is the only place the true cost of a refresh is visible, plus errorMessage on failure.

The gotcha that bites first. Automatic refresh fails silently if the definer role lacks iam:PassRole on itself and a trust policy allowing glue.amazonaws.com to assume it. Manual REFRESH still works, so the view looks healthy right up until the scheduled run never fires. Check this before you delete the pipeline it replaces.

What are the limitations of AWS Glue materialized views?

Source tables must be Apache Iceberg in the same account and Region, incremental refresh accepts only a narrow SQL shape, and the definer role cannot be subject to row or column filters.

Knowing the boundaries saves a painful discovery later, and the full considerations list is worth reading in full.

Hive is disputed. Hudi and Delta Lake are unsupported in every AWS guide. Hive is the interesting case: the Glue guide lists it alongside Iceberg, while Lake Formation and EMR say Iceberg only. If your fact tables are Hive, test first.

Query rewrite is narrower still. The optimizer considers only views within a restricted SQL subset, similar to the incremental refresh rules. Therefore, a complex view may refresh correctly but never get selected. A stale view means Spark silently runs the original plan.

Nested views are documented three ways. The Lake Formation guide describes nested views with refreshes propagating downstream. The Glue and EMR guides say a materialized view cannot reference another. Two of three say no; treat tiered designs as unvalidated.

Eventual consistency is the default. Between refreshes the view is stale by design. For any figure that must be transactionally current, query the base table or refresh manually. Cross-system reconciliation work faces the same staleness decision.

The definer role must be unfiltered. The IAM role creating the view needs SELECT or ALL on every source table with no row, column, or cell filters. You then grant SELECT on the view itself, so consumers query it without base-table access. That is not row or column security on the view: you grant the whole result.

Schema evolution is undocumented. AWS does not say what happens when a base table gains or drops a column. Incremental refresh tracks data changes, not definition changes, so treat schema changes as recreate-and-verify.

Platform requirements are specific. AWS Glue 5.1, Amazon EMR 7.12.0, and AWS-optimized Spark 3.5.6 or later, in roughly twenty Regions. Open-source Spark cannot create or refresh these views. However, you pay twice: for the Iceberg storage the view occupies and the compute each refresh consumes.

How should Glue materialized views change your Power BI or QuickSight strategy?

Push aggregation down into the lakehouse and let the BI layer stay thin.

Most BI teams over a data lake already handle aggregation inside the BI tool through import mode, incremental refresh policies, aggregation tables, or a scheduled SPICE dataset. However, each approach pushes cost and latency into the BI layer while creating a second copy of the truth.

Materialized views move that work back where it belongs. Import models shrink because the source query returns aggregated rows. Power BI DirectQuery models become viable where they were not, because once the model points at the view, the Athena query behind each visual reads a rollup rather than the raw fact table. A SPICE dataset on the view ingests fewer rows. This is pre-computed caching moved from the application tier down to storage, with the catalog handling invalidation.

The governance benefit is arguably larger than the performance one. Rollup logic lives in the catalog, so Power BI, QuickSight, and ad-hoc queries read the same numbers. Teams consolidating onto one lakehouse surface, via OneLake shortcuts in Microsoft Fabric or the Glue Data Catalog, value that over speed.

Should you adopt AWS Glue materialized views?

Adopt them when you have Iceberg tables, repeated aggregation patterns, and dashboards recomputing the same rollups on a schedule, which describes most data lake BI workloads.

Hold off when reports must reflect the base table transactionally, or when your permission model cannot accommodate an unfiltered definer role. On Delta or Hudi you have two routes: migrate those tables to Iceberg, or keep the aggregation in whatever your current engine already provides.

For everyone else, this is one of the rare AWS launches that removes engineering work rather than adding it. The pipeline you never build is the pipeline you never debug. For a second opinion on whether your workload is a fit, reach our team about Athena and Power BI performance tuning, or browse more data engineering case studies.

Frequently asked questions

Q. Do I need to change my Power BI or QuickSight reports?

Usually yes. Query rewrite works only in Spark on Glue, EMR and Athena Spark. BI tools connect through Athena SQL, which does not rewrite, so point the dataset at the view.

Q. Does Athena support creating AWS Glue materialized views?

No. Athena queries materialized views as ordinary Iceberg tables but cannot create, refresh, alter, or drop them. Use Apache Spark in AWS Glue 5.1 or Amazon EMR 7.12.0 for those operations.

Q. How do I refresh a Glue materialized view?

Run REFRESH MATERIALIZED VIEW for an incremental refresh, or add FULL to recompute everything. Outside Spark, call start-materialized-view-refresh-task-run in the AWS CLI. Scheduled refreshes run on their own.

Q. How much does a Glue materialized view cost?

You pay twice: for the Iceberg data the view stores in S3, and for the managed Spark compute each refresh consumes. Query-time cost falls because engines scan the smaller pre-aggregated result.

Q. What is the minimum refresh interval for a Glue materialized view?

One hour for catalog-managed scheduled refreshes. Schedules accept hours, days, or weeks. For anything more frequent, trigger refreshes on demand through Spark SQL or the AWS Glue API.

Q. Do Glue materialized views work with Hive, Delta Lake, or Hudi tables?

No for both. Hudi and Delta Lake are unsupported across all AWS guides. Hive remains disputed: the AWS Glue guide permits it, while the Lake Formation and EMR guides do not.
Test before relying on it.

About ScriptsHub Technologies. ScriptsHub Technologies is a data engineering and analytics consultancy working across the US, UK, and India. Our team builds and modernizes cloud data platforms on AWS and Azure, along with the Power BI and QuickSight reporting layers that run on them. If your data lake dashboards are slower and more expensive than they should be, talk to our AWS and Azure data engineering team.