Once a dbt project crosses 200 models, things break. Six patterns we apply to keep large projects fast and maintainable.
- 01Three-layer architecture: staging → intermediate → marts
Each layer has clear rules. Staging has no joins; intermediate has business logic; marts are flat.
- 02Mandatory model tests at the staging layer
not_null + unique on every primary key + freshness on every source.
- 03Incremental models with surrogate keys
Use dbt_utils.generate_surrogate_key for stability.
- 04CI runs only changed models + downstream
dbt build –select state:modified+ –defer.
- 05Slim CI on a 1% sample of source data
Catches schema changes without full-scan cost.
- 06Documentation as a hard CI gate
Models without descriptions fail the build.
The Slim CI Pipeline (GitHub Actions)
name: dbt slim CI
on: [pull_request]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: pip install dbt-snowflake==1.8.0
- name: Fetch prod manifest
run: aws s3 cp s3://dbt-artifacts/prod/manifest.json target/
- name: dbt build (changed only)
run: dbt build --select state:modified+ --defer --state target/
- name: Upload manifest
if: github.event_name == 'push'
run: aws s3 cp target/manifest.json s3://dbt-artifacts/prod/Ready to optimize your cloud or AI footprint?
Book a free 30-minute architecture review. We will deliver a written cost-and-architecture audit within 48 hours.
Need help with dbt at scale?
Ohveda runs free 30-minute architecture reviews. We will identify your top opportunities in writing within 48 hours — at no cost.