Building a Marketing Mix Model That Cannot Silently Lie

An earlier version of one of my Marketing Mix Modeling systems produced channel allocations that reconciled to the observed total. It also contained a path that could credit a channel with fewer outcomes than the source data had directly recorded for it.

Both statements were true at the same time.

That is the problem with treating a model’s final total as proof that its internal allocations are sound. A system can balance perfectly while violating the evidence inside the balance. Rebuilding this pipeline forced a cleaner distinction between three questions:

Does the arithmetic conserve the outcomes we observed?
Does the model respect reliable evidence that was measured directly?
Is the remaining allocation statistically and causally defensible?

The work described here made the first two questions testable. It did not finish the third. That limitation is part of the result, not a disclaimer added after it.

Define What “Cannot Silently Lie” Actually Means

This is not a claim that the model cannot be wrong. A Marketing Mix Model can have perfect arithmetic and still be wrong because of omitted variables, weak variation, unstable relationships, misspecified transformations, or an overfitted residual allocator.

Here, “cannot silently lie” means something narrower and verifiable:

A channel cannot receive fewer eligible outcomes than were validly observed for it.
Channel outcomes must sum exactly to the eligible observed total for every modeled cell.
Incomplete cohorts, missing feeds, failed assertions, and unsupported estimates must remain visible.
A degraded data source cannot quietly change the baseline output.

Those are pipeline guarantees. They are not causal guarantees.

Treat Governed Click Attribution as a Lower Bound

Privacy loss, blocked scripts, stripped parameters, and broken handoffs remove observable attribution. If a governed attribution pipeline validly records an eligible conversion for a channel, the modeled count should not erase it.

The word governed matters. Tracking can overcount when a conversion event fires on the wrong action, fires repeatedly, includes ineligible records, or is joined incorrectly. I have seen a login event make returning customers look like new session-scoped leads. A faulty event is not a lower bound; it is faulty evidence.

The floor becomes defensible only after the event definition, eligibility rules, deduplication, row population, and time grain match the total being decomposed.

Once that boundary is satisfied, missing attribution is one-directional for that eligible observation set: loss can hide a recorded relationship, but the model should not reduce a relationship already observed.

The Bug Class: Two Guarantees, One Multiplier

The earlier allocation needed two properties:

every channel must remain at or above its observed count;
all channels must sum to the actual outcome total.

The old pattern applied a floor and then normalized all channel values back to 100 percent. Flooring increased the total, so normalization multiplied every channel by a factor below one—including channels sitting exactly on their floor.

Code // inspect before running

raw[channel] = max(raw[channel], observed_share[channel])
total = sum(raw.values())
published[channel] = raw[channel] / total * 100

Check the inputs, permissions, limits, and failure path before using this in production.

The guarantee was destroyed two statements after it was applied.

In the audited output, the later scale factor was below one in every modeled cell. It acted only as a shrink operation. More than a quarter of published rows violated at least one observed floor, and a one-sided validator still reported success while total volume was missing.

The transferable lesson is simple:

If two invariants are enforced by the same arithmetic operation, one may be decorative. Test each invariant after every operation that touches its value.

Give Each Guarantee Its Own Term

The corrected allocation works in outcome counts, not percentage space. The observed portion and the unobserved residual have separate jobs.

Formula // inspect the definition

Observed Evidence Plus an Allocated Residual

Attributed outcomes observed[channel] + share[channel] × residual

The observed term establishes the floor. Only the remaining unobserved outcomes are allocated by the model.

Residual outcomes actual total − Σ observed[channel]

The residual is the exact distance between governed observation and the eligible realized total.

Percentages are derived only after the count allocation is complete. The denominator is never allowed to move underneath the floor.

Nothing multiplies the observed term, so it cannot fall below the floor. The residual shares sum to one, so the allocation reconciles exactly to the actual total.

This also gives the regression a more honest job. Its output is not treated as the complete answer. It weights the distribution of outcomes that remain unattributed after the governed observations are preserved.

Prove the Constraints Are Feasible

The two guarantees can coexist only when the sum of observed floors is no greater than the actual total in every modeled cell. That should never be assumed.

The revised process independently rebuilt both populations and tested every market-month before allocation. No cell was infeasible in the inspected run, and meaningful headroom remained between observed floors and actual totals.

The investigation also exposed two ways to destroy feasibility:

giving the unattributed bucket its own observed floor, which consumes the residual the model is meant to allocate;
constructing channel floors and outcome totals from different row populations or different filter stages.

The second was especially dangerous because both inputs could look reasonable in isolation. The defect existed in their contract.

Validators Must Be Able to Fail

The old validation was one-sided. It detected over-allocation but not under-allocation, and it excluded zero-total cells where severe corruption could hide. It printed a reassuring status instead of establishing the invariant.

The replacement uses independent, two-sided assertions:

independently rebuild eligible actual totals from source data;
independently rebuild each channel’s valid observed floor;
assert that every allocation is at or above its floor;
assert that every cell reconciles to its actual total;
include zero-total, missing, and empty cells;
raise and stop rather than warn and continue.

Then the assertions were tested by deliberately injecting faults. A validator that has never been made to fail is an untested opinion about the code.

Rebuild
A verifier reconstructs ground truth directly from source data without using the implementer's helper functions.
Break
Known floor, reconciliation, zero-cell, and source-contract faults are injected deliberately.
Stop
Every defect must raise a blocking failure rather than produce a warning beside published output.
Compare
Degraded-source paths are checked against the approved baseline at the file level.

The objective is not to make the pipeline pass. It is to prove the pipeline can reject an attractive wrong answer. Generalized from a source-level audit and subsequent implementation notes; identities, raw volumes, paths, and account details are withheld.

Publish Cohort Censoring Instead of Hiding It

In this system, a later conversion is credited to the month in which the originating trial began. Recent trial cohorts are therefore incomplete: some conversions have not had time to occur.

Grossing up a recent cohort can make a chart look complete, but it introduces another modeled number before the underlying outcome is known. An empirical maturity correction was tested and over-recovered against realized outcomes, so the safer release rule was:

reconcile to the outcomes observed so far;
publish an estimated maturity percentage;
flag cells below the maturity threshold;
suppress cost-per-acquisition where the denominator is still materially incomplete.

This keeps an unfinished month from masquerading as a performance decline.

Keep Reporting Transforms Out of the Model

Some spend is purchased under one label while its eventual effect appears under another. Search-engine optimization, public relations, event activity, print, and out-of-home can influence demand without producing a clean click in the month of the invoice.

Moving those dollars inside the regression can manufacture a relationship the time series does not contain. The safer pattern is to finish all outcome allocation first and apply any reporting-only spend reclassification afterward.

That transform should be neutral by construction, but neutrality should still be asserted:

attributed outcome counts must remain identical before and after the reporting transform;
total spend must be conserved for every modeled cell.

Do Not Teach the Model That Data Access Was a Market Event

Adding a second data source partway through a time series can create a false level shift. A simple sum effectively claims the new source contributed zero before collection began and suddenly became valuable on the first available date.

A calibrated backcast can avoid the artificial seam, but its calibration must be local to the boundary. In this implementation, using the entire overlap window distorted a source whose coverage ramped over time. Calibrating on periods adjacent to the seam reduced the boundary artifact across all inspected series.

A separate source audit found a multi-fold jump in an organic-search input at a collection boundary. That kind of break is not safe to hand directly to a regression. Until it is traced to demand, coverage, query mix, extraction, or reporting changes—and the governed historical series is rebuilt—the feature has to remain visibly discontinuous or be excluded. A model will happily assign business meaning to an instrumentation seam.

Two operational defects surfaced during the same work:

an upstream API used a generic client-error response for throttling, bypassing retry logic and dropping a different subset of records across runs;
a plain missing-period calculation could backfill future periods after a feed stopped, disguising an outage as estimated history.

Neither is a modeling innovation. Both can make a model learn a pattern that never happened.

Offline inputs need the same discipline at a different grain. Blending direct-buy cost per outcome across publishers or placements of materially different size can erase the very comparison the model is supposed to support. Preserve publisher, placement, run date, spend, and exposure window before aggregating to the monthly model table.

Prove Fail-Soft Behavior by Hash

A new optional source should not become a new way for the full pipeline to fail—or a reason for its baseline to drift invisibly.

The integration was tested with the source absent, empty, header-only, structurally invalid, and explicitly disabled. Every degraded path was required to produce output byte-identical to the approved pre-integration baseline.

“It still ran” is not a fail-soft test. Identical output, verified by hash, is.

How AI Helped Without Becoming the Authority

AI was useful here because it could inspect the same system through different adversarial lenses, generate fault cases, trace invariants across functions, and challenge a proposed correction faster than one person repeatedly rereading familiar code.

The useful workflow was not “ask an agent whether the model is correct.” It was:

Give the implementer a precise invariant and require a minimal correction.
Give a separate verifier the raw source contract and ask it to rebuild ground truth without reusing implementation helpers.
Ask a methodology reviewer to identify what the arithmetic tests do not establish.
Ask an operational reviewer to simulate missing files, incomplete coverage, throttling, and future gaps.
Define the failure threshold before running the comparison.
Execute the checks on real data and preserve the failing cases.
Require a person to decide whether the evidence supports shipping.

AI accelerated inspection and helped create more ways for the system to fail. It did not supply ground truth, approve the causal interpretation, or make an unverifiable channel claim trustworthy.

This is the same decision boundary I use across marketing automation work: use AI to expand the test surface, not to replace accountable evidence.

Why Build This Instead of Only Using Meridian or Robyn?

Google’s Meridian and Meta’s Robyn are serious open-source MMM frameworks. They should not be dismissed as generic vendor dashboards.

Meridian provides a Bayesian framework with geo-level modeling, adstock and saturation, treatment priors, experimental calibration, reach and frequency inputs, posterior analysis, and budget optimization. Google explicitly describes it as a customizable framework for teams building in-house models. Robyn uses Ridge regression, automated hyperparameter search, Prophet-based trend and seasonality decomposition, response curves, budget allocation, and experimental calibration.

If the decision, data contract, outcome, and validation requirements fit one of those frameworks, starting there is usually more responsible than recreating their statistical machinery. A maintained framework provides documented model behavior, established diagnostics, common outputs, and a wider community capable of finding defects.

The reason to build this system was not that a custom regression is inherently more sophisticated. It was that the measurement problem included operating constraints that existed around the regression:

directly observed outcomes had to remain hard lower bounds;
the remaining unattributed pool had to reconcile exactly to realized outcomes;
trials and paying customers required different timing and maturity treatment;
recent cohorts needed visible censoring rather than a quiet gross-up;
event, print, and out-of-home spend required exposure windows instead of invoice dates;
optional API feeds needed to fail softly without changing the approved baseline;
data-source seams and outages needed to remain distinguishable from market events;
reporting-only spend reclassifications had to be unable to change attributed volume;
validators had to reconstruct ground truth independently from the production helpers.

Those are not arguments against Meridian or Robyn. They are requirements for the surrounding measurement system. A framework can estimate coefficients without knowing which CRM rows are eligible, whether a login event is masquerading as a lead, how a sponsorship should persist across months, or whether two source systems constructed their totals from different populations.

“Build Your Own” Does Not Have to Mean “Ignore the Frameworks”

The most defensible architecture may be hybrid:

Own the data contract, eligibility logic, observation floors, cohort rules, source integrations, and release validators.
Use a custom baseline whose behavior can be inspected end to end.
Run Meridian or Robyn against the same governed modeling table where their assumptions fit.
Compare contribution direction, uncertainty, response curves, residuals, and sensitivity—not just the final channel percentages.
Treat disagreement as an investigation queue rather than choosing the answer that supports the preferred budget decision.

Meridian is especially useful as a Bayesian challenger when geo variation, priors, experiment calibration, and uncertainty are available. Robyn is useful as a regularized, semi-automated challenger with hyperparameter search and built-in trend and seasonality handling. Neither comparison is meaningful if the inputs, controls, outcome definitions, or calibration evidence differ.

When a Custom Build Is the Wrong Choice

Do not build a proprietary model merely to own the code. Use an established framework when the team cannot maintain source contracts, tests, model versions, diagnostics, uncertainty, refreshes, and independent review. Custom code makes every undocumented assumption and implementation defect your responsibility.

This project’s own defects are evidence of that cost. Exact reconciliation concealed a floor violation. A later audit found incompatible feature scales in the contribution calculation. Building the system made unusual constraints expressible, but it did not make the implementation trustworthy by default.

The decision is therefore not custom versus credible. It is:

Which parts of this measurement problem are standard enough to borrow, and which parts are specific enough that the business must define, test, and own them?

What This Work Still Has Not Solved

The revised pipeline can establish arithmetic integrity. The underlying model still needs substantial statistical work:

no time-aware train/test split or holdout has validated prediction;
a reported fit statistic is in-sample and structurally inflated;
a remaining contribution calculation mixes coefficients fitted on standardized features with unstandardized transformed values;
no explicit seasonality term or baseline contribution exists;
several carryover and efficiency constants lack preserved supporting analysis;
outputs are point estimates without uncertainty intervals;
important price, promotion, product, competitor, and market controls remain absent.

The contribution-scale defect is especially important. The observed floors and exact reconciliation now bound the damage: a poor residual weight cannot erase directly observed outcomes or change the total. But the allocation above those floors is not defensible until feature units are corrected and historical results are regenerated and challenged.

The honest status is therefore:

The arithmetic can now be tested independently. The model is not yet validated as predictive or causal.

A Practical Release Gate for MMM Pipelines

Before publishing a channel mix, require separate passes for separate claims.

Gate	Minimum evidence	What it does not prove
Source contract	Stable populations, units, timing, eligibility, deduplication, and lineage	That the variables identify causal effects
Bookkeeping	Observed floors, exact reconciliation, two-sided assertions, and tested failure paths	That the residual split is accurate
Model validity	Time-aware holdouts, residual checks, sensitivity, plausible signs, and uncertainty	That observational estimates equal experimental effects
Business validation	Experiments, shutdowns, known changes, and decision follow-through	That relationships will remain stable forever

Start with How to Build an MMM Without Starting With the Model if the data and decision contract are not written yet.

The model does not earn trust because its percentages look precise. It earns a bounded kind of trust when every guarantee names the evidence it protects, every validator can reject a bad output, and every unresolved limitation remains visible beside the result.