Medallion architecture organises analytical data into layers of increasing quality and business readiness: Bronze preserves permitted source-aligned data, Silver cleans and conforms it, and Gold publishes business-ready data products. The layers define contracts and responsibilities; they do not inherently require three physical databases. Databricks and Microsoft Fabric can both implement the pattern using appropriate storage, processing and governance services. Its value comes from traceable transformations, reusable data and recoverable operations—not from naming folders after metals. Choose physical separation, retention and materialisation according to access, workload and recovery requirements.

Key takeaways

• Preserve source evidence without turning Bronze into an unrestricted, permanent archive.
• Define Silver acceptance rules, quarantine and ordered change handling explicitly.
• Gold can contain detailed facts and dimensions as well as aggregates.
• Delta transactions protect tables; orchestration must manage consistency between layers.
• Implement only the layers and copies that your consumers, security and recovery needs justify.

Contents

• Architecture and flow
• Bronze, Silver and Gold
• Layer comparison
• Databricks implementation
• Fabric implementation
• Delta Lake
• CDC and dimensions
• Performance and cost
• Architecture decisions
• Frequently asked questions

What is medallion architecture?

Bronze–Silver–Gold is a layered, sometimes called multi-hop, lakehouse design pattern. Databricks has popularised and documents it; Microsoft also recommends it for relevant Fabric lakehouse designs. It is a recommended pattern rather than a platform requirement. Azure Databricks’ medallion guidance describes the progression from raw ingestion through validation to business modelling.

Each layer should have an owner, input/output contract, quality policy and recovery path. A customer dataset in Silver might support finance, customer analytics and data science without each team repeating identity cleansing. Gold adds purpose-specific definitions and access. Raw evidence permits reconstruction when logic changes, subject to lawful retention and source availability.

The architecture does not itself validate data or guarantee cross-table atomicity. Those outcomes depend on implementation. A well-operated warehouse with staging and curated marts may already fulfil many of the same responsibilities under different names.

Medallion architecture diagram: end-to-end flow

Text equivalent: ERP, CRM, SQL Server, SaaS APIs, object storage, CSV/JSON and events → Bronze source evidence and metadata → Silver schema, type, key and quality checks → Gold facts, dimensions, KPIs and analytical products → Power BI, Databricks SQL, enterprise analytics, data science and approved AI/ML. Failed checks enter quarantine; corrected logic or data follows controlled replay. Orchestration, lineage, security, monitoring, CI/CD and cost controls span the path. Arrows are logical dependencies, not instructions to copy every dataset three times.

Planning a Modern Enterprise Lakehouse?

Start with consumer needs and acceptance contracts. Discuss Your Lakehouse Architecture before deciding how many physical layers to create.

Bronze layer: raw, source-aligned ingestion

Bronze preserves permitted source fidelity and change evidence. Add provenance such as source file, extraction window, record/event identifier, source sequence, source timestamp, ingestion timestamp and run ID. Append/history patterns often support replay, but explicit retention and deletion obligations still apply. Avoid destructively cleaning the only recoverable copy.

For semi-structured sources, retain the original payload or durable source file alongside parsing metadata. A malformed JSON line should become an observable record with a reason, not disappear because parsing failed. Keep schema drift visible; an unknown field is different from a required field changing type. Auto Loader is one Databricks ingestion option, but its schema/checkpoint handling must be configured for the source and runtime.

Synthetic order payload:

{"event_id":"E102","order_id":"O1","sequence":"2",
 "op":"upsert","customer_id":"C1","product_id":"P1",
 "order_date":"2026-10-01","quantity":"3","unit_price":"25.00"}

The accompanying envelope records `source_file`, `source_position`, `ingested_at` and `run_id`. Sequence is the source’s order, not arrival time. The original strings remain available even if conversion fails. Restrict Bronze readers: preserving raw detail increases sensitive-data exposure, storage and retention responsibilities.

Silver layer: cleaned and validated data

Silver applies explicit types, required-field checks, normalisation, deterministic deduplication and conformed business keys. Validate customer/product references and distinguish unresolved references from genuinely optional values. Do not invent missing identifiers or silently convert invalid amounts to zero.

The example becomes order O1, sequence 2, quantity 3, decimal unit price 25.00, amount 75.00 and date 2026-10-01, with source provenance retained. Silver assertions include positive quantity, permitted operation, nonnegative supported price, a valid date, unique current order key and known customer/product IDs. Refunds require a separate business contract rather than assuming every negative amount is invalid in every domain.

Deduplicate transport copies, then resolve change ordering before publishing a current entity. Equal key/sequence with different payloads is a conflict to investigate. The latest invalid change must not be silently replaced by an older valid version, making stale data look current. Choose an explicit hold/suspend policy and expose freshness status.

Quarantine stores the original record, reason, rule version and repair ownership with controlled access. Reconcile accepted, rejected, duplicate, superseded and deleted outcomes to the input. Silver should remain reusable at useful detail; avoid embedding one dashboard’s definitions into the shared contract.

Gold layer: business-ready data products

Gold defines business entities, detailed fact tables, dimensions, star schemas, KPI datasets and justified aggregations. An order-line fact and customer dimension are Gold products even when they are not aggregated. A semantic model adds relationships, DAX measures and consumption security; it is not identical to a Delta table.

In the shared synthetic example, current accepted orders O1 and O2 contribute 75.00 and 40.00. The Gold daily summary is:

This is a current-order teaching model, not an accounting ledger. It excludes quarantined records and honours a delete tombstone. Finance, tax, refunds, currency conversion and historical customer attribution need separately owned rules. Publish only after the expected reconciliation and business acceptance gate.

Power BI can consume an appropriate semantic model, while Databricks SQL can query curated tables. Data science may use Silver or Gold depending on permitted detail and purpose. AI/ML features also need point-in-time correctness; a current customer attribute must not leak future information into historical training examples.

Bronze vs Silver vs Gold

The distinction is responsibility, not a confidence percentage. Gold is fit for a defined purpose, not automatically correct or safe for every use.

Medallion architecture in Azure Databricks

An Azure implementation can use ADLS-backed storage, Delta tables, Unity Catalog for governed objects, Spark/PySpark transformations and Databricks SQL consumption. Organise a domain catalog with Bronze/Silver/Gold schemas when that matches ownership and access. Separate catalogs or pipelines when stronger administrative or operational boundaries justify them; naming alone is not isolation.

Text equivalent: approved sources/ADLS → Auto Loader or selected connectors → Bronze Delta → Spark/Lakeflow Silver plus quarantine → Gold facts/dimensions → Databricks SQL/Power BI/approved ML. Unity Catalog, orchestration, event logs, permissions and CI/CD apply across the flow.

Current documentation calls the product Lakeflow pipelines, built on Apache Spark Declarative Pipelines with Databricks extensions. Older material may say Delta Live Tables or Lakeflow Spark Declarative Pipelines. The DLT transition guidance says existing DLT code remains supported, while recommending updated Python names. This is not a mandatory rewrite simply because terminology changed.

New pipeline code uses `from pyspark import pipelines as dp`; streaming tables and materialized views have distinct decorators. Some capabilities, including expectations and AUTO CDC, are Databricks extensions rather than universal open-source Spark APIs. Review the Python reference against the selected Azure runtime. Dataset functions return DataFrames; avoid external writes or actions inside those declarations.

Step-by-step Databricks pipeline example

Use the supplied synthetic order events, customers and products. Ingest original records into an isolated Bronze demo table with an explicit schema/provenance envelope. Parse without discarding malformed input, validate sequencing, collapse identical transport duplicates and quarantine conflicts. Resolve the latest event per key before evaluating current business validity, while retaining tombstones. Conform references and publish accepted Silver candidates. Build an order fact, customer/product dimensions and daily KPI. Schedule only after quality gates, then monitor event outcomes and practise replay.

This abbreviated declaration demonstrates the current API against an already prepared demo candidate table; it is not a complete production CDC pipeline:

from pyspark import pipelines as dp

@dp.materialized_view(name="_medallion_demo_gold_orders")
@dp.expect_or_fail("positive_qty", "quantity > 0")
def demo_gold_orders():
    return spark.read.table("_medallion_demo_silver_orders")

This declaration exposes accepted detailed rows; the companion SQL provides daily aggregation. Expectations can record, drop or fail depending on the chosen policy. Dropping invalid records does not automatically preserve their payload in quarantine. Build and reconcile that route explicitly.

The companion plain-PySpark batch example separates parsing, state and acceptance and contains writes disabled by default. Its expected output is two accepted current orders, one tombstone, three quarantined inputs, one identical duplicate and two superseded events from nine input records. Local reference tests check those outcomes; Fabric/Databricks runtime execution remains required. Lakeflow best practices distinguish streaming tables, materialized views and temporary views by purpose.

Medallion architecture in Microsoft Fabric

OneLake, Lakehouse Delta tables, Data Factory pipelines, Dataflow Gen2 and Fabric Spark notebooks provide relevant implementation choices. Use a Warehouse for justified relational Gold requirements; SQL analytics endpoints and Power BI semantic models provide distinct serving surfaces. Fabric medallion guidance describes physical deployment options, while its Lakehouse tutorial demonstrates a source-to-consumer scenario rather than a universal production reference.

Separate workspaces can strengthen administrative boundaries but increase promotion, access and orchestration work. A single schema-enabled Lakehouse can express logical layers; it does not automatically prevent broad workspace contributors from reaching raw data.

Text equivalent: Data Factory/Dataflow/notebook ingestion → OneLake Bronze → supported Spark/SQL cleansing and quarantine → Gold Lakehouse or Warehouse → semantic model/Power BI. A materialized-lake-view route is optional and must meet support requirements. Workspace permissions, lineage, capacity and deployment controls span the design.

Step-by-step Microsoft Fabric implementation

Create an isolated capacity-enabled workspace and schema-enabled Lakehouse; approve connections and least-privilege identities. Land the same fixture and capture provenance. Use the shared notebook’s schema, sequence and quality rules to create Silver candidates. Validate reconciliation, then model Gold and, where justified, load a Warehouse through a supported path. Configure the semantic model and selected storage mode. Orchestrate ingestion before transformations/publication; monitor Spark, refresh outcomes, access and capacity cost.

For supported declarative processing, this Spark SQL example assumes the demo Silver table has already passed its acceptance gate:

CREATE MATERIALIZED LAKE VIEW IF NOT EXISTS
  dbo._medallion_demo_gold_sales_by_day AS
SELECT order_date, COUNT(*) AS accepted_orders,
       SUM(amount) AS revenue
FROM dbo._medallion_demo_silver_orders
GROUP BY order_date;

Run DDL only in an approved demo environment. The materialized lake views tutorial demonstrates Bronze–Silver–Gold dependencies in a schema-enabled Lakehouse. These views persist results; they are not ordinary nonmaterialised SQL views.

As checked in October 2026, Microsoft marks PySpark MLV authoring as Preview, with full-refresh-only behaviour. SQL-defined views can use optimal refresh, but it is not a promise that every query refreshes incrementally. Delta CDF, supported constructs and data-change patterns matter; updates/deletes need compatible identity hints for incremental processing or can trigger a full refresh. See MLV authoring and refresh rules. Do not transplant Databricks `dp` decorators into a Fabric notebook.

Need Help Building Bronze, Silver and Gold Data Pipelines?

Discuss your lakehouse requirements to define a bounded implementation, quality gates and recovery tests.

Databricks vs Fabric for medallion architecture

Both support the pattern. Select by requirements and existing operations; no universal faster/cheaper claim follows from this table. Detailed platform selection belongs in a separate comparison, linked only after publication.

How Delta Lake supports medallion architecture

Delta combines Parquet data with a transaction log, supporting table transactions, schema checks, versioned reads and change operations. Batch read/write documentation explains schema and time-travel behaviour. Medallion is a design pattern; Delta is a table technology. Neither requires the other conceptually, although both platforms commonly combine them.

MERGE can apply inserts and updates, but source uniqueness and event ordering remain necessary. Delta’s optimistic concurrency detects relevant write conflicts; it is not a global transaction across every Bronze, Silver and Gold table. Publish consistent multi-table versions through an agreed gate or manifest.

Time travel is not an unlimited backup. VACUUM and retention affect which historical files remain readable. Set retention beyond agreed replay/reader needs where lawful, test restoration and maintain separately justified backups. Do not disable safety protections or prescribe a universal low retention interval.

ETL, ELT and medallion architecture

ETL transforms before the final analytical load; ELT transforms after landing in the analytical platform. Layered lakehouse processing often fits ELT, but pre-load minimisation or standardisation can still be necessary. Medallion is not exclusively ELT. Batch, streaming, incremental delivery and CDC describe different dimensions of the pipeline. Orchestration must order accepted dependencies, not merely launch three jobs at a fixed interval.

Data quality and governance across layers

Use measured counts and denominators rather than arbitrary confidence percentages. Dataset endorsement/certification should follow an explicit organisational review, not imply an external certification. Track the rule and source version behind a quality scorecard. Restrict quarantine as carefully as Bronze; error payloads can contain sensitive information.

Retain operational lineage: run ID, inputs, source window, target versions, rule version, acceptance decision and owner. Audit access and monitor schema changes. Product permissions, not layer names, enforce boundaries; Fabric security is one platform-specific reference.

Incremental processing, CDC and slowly changing dimensions

Incremental ingestion processes a bounded change set; a watermark is progress, not proof that late records cannot exist. CDC carries operations and ordering; Delta CDF describes table changes and is not automatically the original source’s CDC protocol. Preserve unique event identity, key and source sequence. Apply current-state updates only when incoming order supersedes stored order; equal-sequence conflicts require review.

Retain a tombstone or equivalent sequence ledger so a late pre-delete event cannot resurrect a deleted entity. SCD Type 1 updates the current attribute. SCD Type 2 preserves effective history with nonoverlapping intervals and a stable business key plus version identity.

For customer C1, an update changes region North to West effective 2026-10-02. Bronze keeps both events. Silver validates their order. Type 1 returns West; Type 2 retains North for `[2026-10-01, 2026-10-02)` and West thereafter. A sale on 1 October uses North under an as-of attribution policy. A late earlier event may require rebuilding the affected key’s history, not appending a second overlapping current row.

Databricks AUTO CDC supports sequence-aware processing and SCD choices. It requires an appropriate Lakeflow configuration; do not assume it exists in ordinary Fabric Spark or open-source SDP. Watermark expiry, duplicate source matches, missing deletes and checkpoint loss all need replay tests.

Medallion architecture performance optimization

Measure file counts/sizes, scan selectivity, shuffle, skew, job duration, consumer latency and concurrent workloads. Compact small files when justified; choose partitioning or clustering from actual access and change patterns. Avoid high-cardinality partition schemes and universal file-size prescriptions. Incremental transformations can reduce work but increase state/recovery complexity.

Fabric V-Order is a platform optimisation with a write/read trade-off, not a generic Delta requirement; verify workspace profile defaults. Databricks liquid clustering and supported automated maintenance have their own runtime/table prerequisites. Test maintenance alongside ingestion rather than letting compaction compete unchecked with reporting.

Medallion architecture cost and TCO

Monthly TCO = ingestion + billable layer storage + batch/streaming transformations + maintenance + orchestration + monitoring/governance services + BI/SQL consumption + engineering/support. Allocate shared capacity once. Add separate workload charges only where applicable; no cloud price or ROI is invented here.

The companion calculator uses editable monthly ingest volume, source count, retention, output ratios, pipeline frequency, compute per run, reprocessing, streaming hours and report concurrency/serving hours. Rates start blank. Example workloads are planning assumptions, not measurements. A simple model estimates Bronze stored GB as monthly landed GB × retention days/30, then separately estimates Silver/Gold and version history. It assumes steady ingestion and does not account automatically for compression, bursts, deleted-file retention or replicas.

Three physical layers can improve reuse and replay while adding storage and transformation cost. Fewer materialised stages or temporary views can be sensible when recovery and governance remain sound. Compare the complete alternative, including upstream processing and maintenance; do not assume three layers cost exactly three times as much or always save money.

Real-world medallion architecture examples

These are hypothetical applications, not delivered customer case studies.

Common medallion architecture mistakes

Medallion architecture vs a traditional warehouse

Traditional warehouses already use staging, conformed entities, facts/dimensions and marts. Medallion names quality/reuse boundaries, often in a lakehouse handling structured and semi-structured data with separate compute/storage choices. It complements star-schema modelling rather than replacing it. Governance, BI and ML access still need design in either approach; moving files into a lakehouse does not automatically modernise the business model.

Is medallion architecture always necessary?

Data mesh is an ownership/product operating model, not the same decision as Bronze/Silver/Gold. Use fewer or more logical stages when justified, and materialise only where cost, access or recovery benefits warrant it.

Production implementation roadmap

How Prominent AI Services LLP can help

Prominent’s verified pages describe data engineering, Microsoft Fabric implementation, Databricks lakehouse engineering and Power BI development. Related Azure and SQL Server/Azure SQL services provide relevant integration context.

Scope assessment, architecture, a bounded proof of concept, implementation, acceptance and ongoing optimisation/support. Agree deliverables for quality, Delta maintenance, pipelines, warehouse modernisation and consumption rather than presuming every capability is included. No certification, numerical outcome or customer endorsement is implied.

Frequently asked questions

Conclusion: operate the contracts, not the colours

Medallion architecture is useful when each layer has a clear purpose, owner and recovery contract. Preserve evidence, validate reusable entities and publish accepted business products without unnecessary copies.

Discuss Your Lakehouse Architecture to define an appropriate assessment or implementation scope.