The daily feed, versioning, and freshness

Financial data reaches Heysen through a daily feed from the agency’s legacy commission system. This page explains the delivery contract in operator terms: what arrives, how to tell whether it is complete, and what the platform refuses to do.

What arrives

A daily delivery contains eight datasets:

Dataset	Contains
agents	the people in the source system
customers	policy owners and other client records
policies	the policies themselves
policy-agent-mappings	who is attached to a policy
policy-customers	policy-to-customer links
commission-results	commission results, by layer
settlements	legacy settlement records
reconciliation-controls	the controls used to verify the delivery

A delivery can be read over the API or over SFTP. Both routes lead to the same source of truth; the important part is that you pick one delivery and read it completely.

Pick a delivery by what it says about itself

A source directory can hold several complete packages. Do not decide which one is newest from directory order, file modification time, or the filename. The package declares its own reference date and the time it was actually observed — use those.

Each delivery has an identifier and a manifest. Read the manifest, then read every page it lists, then verify. A package that is half-read is not evidence of anything.

Every record is versioned

Source records are append-only. The same business record can appear more than once, distinguished by a version number. “What this record says now” is the highest version, not the sum of everything on the table.

This rule is not theoretical. The source used to deliver exactly one row per record, and every consumer query was written without a version filter — which was harmless, right up until a delivery appended second versions of some records. At that point, every query that summed or counted across the table counted the old and the new row, and commission totals for the affected agents doubled.

The fix is a single rule enforced everywhere: read the current version only. The verification tooling already used it. The consumers had to be brought in line, and a guard now fails the build if a new query touches a commission table without it.

The same trap exists outside the money path. Counting policies without a version filter makes a legitimate policy look like it has two candidates, and the platform then refuses to auto-map it. The rule applies to any table in the replica that can hold multiple versions.

The reconciliation gate

Before a delivery can be read, three independent checks run:

1. Per-dataset count and content hash — the number of rows and their canonical digest must match what the manifest claims. This catches transport damage.
2. Duplicate detection — a delivery that carries the same record identifier twice is rejected.
3. Per-group row and amount controls — for each group of results, the row count and the money fields must reconcile. This catches the case where the file is intact but the source computed it wrong, which the first two checks cannot see by construction.

If any check fails, the import does not advance the checkpoint. The previous complete batch stays in place, labelled with its own timestamp, and operators see the failure code — CONTROL_MISMATCH for a controls failure.

Caution

No partial imports. A delivery that fails verification is rejected as a whole. The platform will not import the datasets that passed and skip the one that did not, because a partially-imported batch is a data set that looks complete and is not.

Other failure modes are handled as explicit outcomes rather than best-effort merges: out-of-order deliveries, a record version that goes backwards, the same version arriving with different content, missing rows, and mixed batches. Each produces a defined result; none of them is allowed to end in “imported successfully”.

Freshness: a bounded window, not an unlimited claim

A daily feed is valid for a bounded window — currently 24 hours for the financial data. You cannot extend that window by re-compressing a package or by editing its generation timestamp; the window is about when the data was observed, not when the file was written.

If no valid new package arrives inside the window, the platform says “data update delayed”. It does not keep presenting yesterday’s data as current, and it does not need anyone to fabricate the days in between — the next complete package restores normal operation.

Qualification observations are stricter still: they describe a moment, not a state. The observation file that ships with a delivery is historical — the platform labels it “frozen/deferred as of time” and never restates it as live eligibility, as already paid, or as received. Its reference date is the day the source ran, not a payment date.

Note

A delivery grants no payment authority. Its payment_authorization flag is explicitly not granted, which is a statement about what the file permits — not a statement that an agent’s income is zero. Never read the two as the same thing.

Money handling

• Amounts are handled as decimal / fixed-point values, never as floating point.
• The platform preserves what the source sent: amount strings, negative values, true zeros, and nulls.
• A null is not a zero. Where a value is unknown, the platform shows it as unknown. Printing 0 for an absent value turns “we don’t know” into “there is nothing”, and those lead to opposite decisions.
• Money is reconciled per currency and per status, not as one blended total.

Re-reading is safe

Re-reading the same delivery is expected and idempotent: it adds zero new source rows, changes no amounts, and leaves operator decisions untouched. A delivery whose financial facts are unchanged can legitimately keep the same identifier — a fresh observation is not a fresh commission.

Manual policy-to-agent mappings live in their own layer. Refreshing source data never overwrites a mapping a human confirmed.

What operators compare

When reconciling, compare current records — one per record, the highest version — with the source’s declared row count. The replica also keeps the historical rows; that total is larger by construction and will never match the source’s count. Comparing the wrong column produces a phantom discrepancy.