What a reconciliation gate actually protects

Every daily delivery we import carries a manifest, and the manifest makes claims: this dataset has this many rows, this is its content digest, and these groups of results add up to these amounts. The importer’s job is to disbelieve all of it until it has checked.

The obvious checks are the ones people usually write first. Row counts match. Content hashes match. Good — that catches a truncated download, a corrupted transfer, a file that was assembled from two different days. Necessary, but nowhere near sufficient.

The gap

Counts and hashes verify that what arrived is what was sent. They say nothing about whether what was sent is internally correct.

Those are genuinely different failure classes:

• A transfer that loses bytes fails a hash check. A hash check is exactly right for this.
• A source that computes a group’s total from the wrong set of rows produces a perfectly intact file with a wrong total. Every hash matches. Every count matches. The number is wrong.

No amount of transport-layer verification reaches the second case, because from the transport layer’s point of view nothing is wrong.

Group controls

So the delivery also carries per-group controls: for each group of results, the row count and the money fields, declared independently of the data itself. The importer recomputes the same aggregation from the rows it just received and compares, group by group, in both directions — a group with data but no control, and a control with no data, are both failures.

That catches the third case that neither counts nor hashes can: the file is intact, the rows are all there, and the arithmetic the source performed does not agree with the arithmetic its own rows imply.

When a control does not reconcile, the import stops and records the failure code. Nothing is written.

Why it fails closed, all of it

The tempting alternative is to import the datasets that passed and hold back the one that failed. It is tempting because it keeps the UI fresh, and it is wrong because a partially-imported batch is indistinguishable from a complete one at the point of use. Rows exist. Counts look plausible. Nothing on the surface says one dataset is a day behind.

So the gate rejects the whole delivery. The checkpoint does not advance, the previous complete batch stays in place labelled with its own time, and the next scheduled run retries from the same checkpoint. Operators see the error code rather than a subtly inconsistent dataset.

The same fail-closed logic covers the smaller cases: a delivery that carries the same record identifier twice, a record version that goes backwards, the same version arriving with different content, an empty dataset where the local copy still has rows. Each has a defined outcome, and none of them ends in “imported successfully”.

Idempotency is a separate property

Re-reading a delivery must be safe, because retries happen. After a successful import, re-reading the same delivery adds zero new rows and changes no amounts — a check we run deliberately, not hope for.

But note what that proves: writes do not duplicate. It says nothing about whether reads are correct, which is a distinct failure with a distinct signature. We learned to keep those two checks separate, and to stop treating a green replay as evidence about the read path.

What we would tell another team

• Store the controls in the delivery, not in a separate channel. A control that arrives separately can be a day out of step with what it describes.
• Reconcile in both directions: data without controls, and controls without data.
• Reject the whole batch. Never partially apply.
• Advance the checkpoint last, inside the same transaction as the data write, so a failure leaves you exactly where you were.
• Distinguish “the transfer was good” from “the data is good” in your logging. A failure that reports which of the two it was saves an hour of guessing.