How We Scaled a Data Platform to Tens of Isolated Products with a Single Terraform Module

A template-driven data product model that delivers fully isolated environments (storage, compute, governance, and access) through one shared module instantiated via configuration.

Tags: Databricks, Data Platform, Unity Catalog, Infrastructure as Code, Data Mesh, Swiss Enterprise

Context

The cloud platform was live: governance, networking, identity, all running as code on a dedicated Azure tenant. Now the challenge shifted: multiple business units within the same Swiss client needed independent data environments. Each team required their own storage, compute, secrets management, and access control, completely isolated from every other team. And the platform had to scale from the first data product to many without proportional growth in engineering effort.

The answer was a template-driven data product model on Databricks, built entirely as Infrastructure as Code. Every resource, from storage accounts to Unity Catalog entries to budget policies, defined in Terraform and provisioned through a single pipeline.

One product, fully isolated

Each data product is a self-contained environment. When the platform provisions a new product, it creates a complete set of dedicated resources, no sharing, no shortcuts.

From one to tens: template-driven scaling

The key architectural decision was making every data product an instance of the same Terraform module. Adding a new product means creating a single configuration file (approximately one hundred lines) that references the shared module with product-specific parameters: a name prefix, three static IP allocations for private endpoints, and the environment. Everything else, storage, Key Vault, service principal, Unity Catalog, compute policies, Entra ID group, Azure DevOps project, is provisioned automatically.

1. Configuration file created
2. Entra ID group provisioned
3. Azure DevOps project created
4. Storage + Key Vault + private endpoints
5. Unity Catalog + compute policies
6. Team notified: product ready

Tens of data products are running in production today, all provisioned through this pattern. The project team did not write separate configurations for each; they wrote one module and instantiated it through configuration. When a security improvement is made to the module, every product benefits on the next apply.

The best infrastructure patterns are the ones where adding capacity is a configuration change, not an engineering project. When a new business unit needs a data product, the answer is a pull request, not a project plan.

Every change audited, every resource private

Isolation between data products is identity-based. Each product has its own Unity Catalog, its own Entra ID group, its own service principal, and its own RBAC assignments. There is no implicit sharing; accessing another product's data requires an explicit consumer grant defined in code. Storage accounts and Key Vaults sit behind private endpoints on the shared platform network, with zero public access.

Changes follow the same discipline as the cloud platform: Git commit, pull request, peer review, Trivy scan, Terraform apply. Adding a consumer group that grants one team read access to another team's catalog is a code change, reviewed, approved, and auditable. Removing access is the same process in reverse.

From infrastructure to machine learning

Because the cloud platform and data product infrastructure were fully automated, the team scaled quickly beyond core data engineering. Machine learning pipelines were built using structured frameworks. Lightweight integration APIs were built on serverless functions instead of Databricks, right-sizing compute for workloads that did not need Spark. Consumer service principals enabled cross-unit analytics; business units that previously had no way to share data could now do so through controlled, auditable access grants.

The dedicated tenant, originally a pragmatic workaround for missing permissions, became the enabler for this speed. With full control over identity, networking, and governance, the project team could ship new capabilities without waiting for corporate IT approvals. The only delays came from traditional IT processes where automation and lean ways of working were not yet established, never from the project team.

Outcome

The data product model proved that infrastructure scaling does not require proportional engineering effort. A single well-designed module, instantiated through configuration, delivered tens of fully isolated environments, each with enterprise-grade security, governance, and cost controls. The platform continues to run in production and has become the foundation for the client's analytics and machine learning initiatives.

This is the type of engagement our Data Platform on Databricks use case covers. The cloud platform foundation described in our companion article made this possible. Whether you need the full stack or just the data layer, we have built and operate platforms like this in production.