Merge rules
How a waterfall picks each field's value when vendors disagree, records provenance and conflicts, and builds one golden record across stages.
Before you start
Every hit in a stage is a merge candidate: its fields, the vendor, its confidence, whether it was
verified, the endpoint's quality score and the order it ran in. Misses are not candidates. For each
field that any candidate filled, the stage picks one value with that field's strategy, set in
merge.fields, or merge.default when the field has none.
Strategies
| Strategy | Picks | Ties broken by |
|---|---|---|
first_non_null | The first filled value in step order | — |
highest_quality | The endpoint with the highest quality score | confidence, then verified, then order |
prefer_verified | A verified value | confidence, then order |
most_recent | The newest value by merge.recency_field (default updated_at), or when the attempt finished | order |
consensus | The value most vendors agree on, compared case- and whitespace-insensitively | the best-quality candidate among the tied values |
union | Every distinct item across vendors, flattened from arrays, in step order | — |
Worked example
A stage runs three vendors with all_then_merge, merge.default: "highest_quality" and
merge.fields: { "email": "consensus" }:
| Vendor | Quality | email | title |
|---|---|---|---|
| A | 0.82 | ada@example.com | CTO |
| B | 0.91 | ADA@example.com | Chief Technology Officer |
| C | 0.77 | a.lovelace@example.com | (empty) |
emailusesconsensus. A and B agree once case and spaces are ignored, so they win 2 to 1, and B is kept because it has the higher quality. C goes in the conflicts list.titleuses the defaulthighest_quality, so B is picked. A'sCTOis a conflict. C didn't fill the field, so it isn't a candidate.
{
"data": { "email": "ADA@example.com ", "title": "Chief Technology Officer" },
"_provenance": {
"email": {
"provider": "b", "strategy": "consensus", "confidence": 0.9, "verified": true,
"conflicts": [{ "provider": "c", "value": "a.lovelace@example.com" }]
},
"title": {
"provider": "b", "strategy": "highest_quality",
"conflicts": [{ "provider": "a", "value": "CTO" }]
}
}
}Each provenance entry also carries endpoint_id, call_id, stage_id, step_id and fetched_at.
The conflicts list holds up to 20 disagreeing values; union fields have none. Show conflicts to a
reviewer rather than dropping them.
Across stages: the golden record
A pipeline merges inside each stage, then builds the run's data from its stages in order. The
first stage to fill a field keeps it; later stages only add fields that are still empty. Put
the stage you trust most for a field first, or use fields on a later stage so that stage returns
only what it's there to add:
{ "id": "company", "capability": "company.enrich", "fields": ["company_name", "industry", "employee_count"] }Later stages read earlier ones in two ways. Capability inputs fill from the golden record by name.
input_map reads a specific value with stages.<stage_id>.<field> or input.<field>. A when
on a stage or step can test golden-record fields, including _verified and _confidence.