You will build and operate the data backbone of ODL: bulk and streaming ingestion from legacy source systems, medallion-layered storage (Bronze\/Silver\/Gold), identity resolution and golden-record consolidation, source-to-canonical mapping and crosswalks, and the data-quality and reconciliation gates that prove data is complete and correct before it is published. This is the volume engine of the program \u2014 every new client onboarded flows through the pipelines you build.<\/p>\n\n\n\n
What You\u2019ll Do:<\/strong> What We\u2019re Looking For:<\/strong> Bonus Points:<\/strong>
\u25cf Build batch-seed and event-tail ingestion per source system, including seed\u2192tail watermark hand-off, idempotent upserts, and dedup ledgers
\u25cf Build and operate medallion layers with reprocess-from-Bronze, pipeline orchestration (checkpoints, retry\/backoff, DLQ), and full observability.
\u25cf Build data-quality gates (quarantine \/ pass-with-flag), quality scoring, and a reconciliation engine covering count, record, and financial reconciliation \u2014 financial is zero-tolerance.
\u25cf Build identity matching combining deterministic rules with probabilistic scoring and confidence bands; deliver deduplication, golden-record materialization, and survivorship rules, calibrating match thresholds with labelled data.
\u25cf Author and maintain source\u2192canonical structural mappings and value crosswalks (e.g., collapsing 1,800+ raw employment-status values to ~20 standard ones) as governed, versioned configuration
\u25cf Enforce data contracts at the boundary: schema registry, fail-fast validation, and semver-compatible schema evolution.<\/p>\n\n\n\n
\u25cf 5+ years building production data pipelines at scale.
\u25cf Kafka depth: consumers\/producers, replay, DLQ, exactly-once \/ idempotent processing patterns.
\u25cf Strong SQL and solid ETL fundamentals.
\u25cf Java and\/or Python in production.
\u25cf Medallion \/ lakehouse layering, CDC, watermark\/checkpoint patterns, and batch\u2013stream hand-off.
\u25cf Data-quality frameworks: validation rules, quarantine and re-entry, quality scoring, reconciliation.
\u25cf Entity resolution \/ MDM exposure: record matching, dedup, survivorship \u2014 via commercial tools (Informatica MDM, Reltio) or custom builds.
\u25cf Data mapping and crosswalk discipline: profiling messy datasets, authoring governed reference data, config-as-code (YAML\/JSON, Git).<\/p>\n\n\n\n
\u25cf Probabilistic record linkage at depth \u2014 blocking\/candidate generation, scoring models, threshold calibration (expected at senior level).
\u25cf Schema registry experience (Avro\/Protobuf).
\u25cf Extracting from mainframe or older RDBMS sources with limited CDC support.
\u25cf Financial reconciliation in finance-adjacent domains.
\u25cf Benefits administration or healthcare domain knowledge.<\/p>\n\n\n\n