DAVA Dedup

Quickstart (Python)

bashpip install dava-dedup

pythonimport asyncio
from dava_dedup import Client

async def main():
    async with Client(api_key="dava_live_…") as c:
        with open("customers.csv", "rb") as f:
            data = f.read()
        cols = await c.list_columns("customers.csv", data)
        keys = [name for name in cols if "email" in name.lower()]
        result = await c.find_duplicates("customers.csv", data, key_columns=keys)
        print(f"{result.duplicate_rows}/{result.rows_total} rows "
              f"in {len(result.clusters)} clusters")
        for cluster in result.clusters:
            canonical = next(m for m in cluster.members if m.is_canonical)
            print(f"  cluster {cluster.cluster_id}: keep row {canonical.row_index}")

asyncio.run(main())

Quickstart (TypeScript)

typescriptimport { Client } from "@avaresearch/dava-dedup";

const c = new Client({ apiKey: process.env.DAVA_API_KEY! });
const cols = await c.listColumns("customers.csv", file);
const result = await c.findDuplicates("customers.csv", file, ["email"]);

How clustering works

For each row, Dedup builds a normalized key from the chosen key columns (lowercased, whitespace-collapsed, non-alphanumerics stripped for text fields). Rows with the same key form a cluster. The canonical row is the one with the fewest empty cells; ties broken by lowest row index.

The full Dedup pipeline (later release) adds blocking, embedding-based fuzzy match, and survivorship rules. The current SDK + API surface is the deterministic preview.

AI adjudication and its limit

Pairs that score close to the merge threshold — but not over it — fall into a borderline band. When a profile enables AI adjudication, those pairs are sent to a model for a second opinion, most-marginal first.

Adjudication is capped per job. The default limit is 30 pairs, and the limit is a cost control, not a quality judgement: pairs beyond it are returned as unadjudicated and left unmerged without being reviewed. That is deliberately distinct from below_threshold (the rules were confident) and from ai_reject (a model looked and declined).

Because the limit is per job rather than per row, widening the band does not increase cost — it increases the share of the band nobody examines. Every result reports adjudication.borderline_pairs_found, borderline_pairs_adjudicated and borderline_pairs_unadjudicated so the coverage is always visible, along with a projected token cost for reviewing the remainder.

To see which pairs were never reviewed, run with a profile. The job then writes borderline_pairs.csv (download it with GET /v1/jobs/{id}/download-url?artifact=borderline_pairs): one line per borderline pair with its two row numbers, score, and a status of merged_by_ai, kept_apart_by_ai or not_reviewed. The annotated file gains a dava_unreviewed_pairs column counting each row's unreviewed pairs. Without a profile the scan stops at the limit, so only the counts above are available (scan_truncated says so).

Raise the limit for a single job with adapter_options.ai_max_pairs. Set it to 0 to disable adjudication entirely.

API surface