Distributed SQL Failure-Semantics Evidence Director

The mandate

A distributed SQL pilot intended to replace three regional settlement databases remained available during fault tests, yet replay found ledger entries whose commit order, visibility and retry outcome could not be explained consistently. Vendor dashboards declared zero data loss while finance controls found duplicate adjustments and temporarily impossible balances. The defined problem is to determine whether the platform’s actual failure semantics satisfy the settlement control model before cutover.

The named deliverable is a Distributed SQL Failure-Semantics Evidence Dossier and Cutover Recommendation. It must describe transaction and consistency assumptions, map application behaviours to database guarantees, provide a fault-injection plan and results, reconcile observed histories, identify configuration and code dependencies, define compensating controls, and state whether to proceed, constrain scope or reject the target design.

Three milestones govern twelve weeks: by the end of week three, approve the transaction inventory, invariants and evidence baseline; by week eight, complete partition, clock, lease, replica, retry and region-loss experiments with reconstructed histories; and by week twelve, deliver the signed opinion, cutover conditions, rollback triggers and remediation backlog. Payment follows each accepted milestone.

The settlement risk panel and finance controller decide acceptance. The dossier passes only when every test is repeatable from versioned scripts, observed histories can be reconciled to named invariants, unknown outcomes and client retries are treated explicitly, vendor claims are separated from client evidence, and a dry-run cutover decision can be made without the consultant interpreting the model in the room.

The client will provide schema, transaction traces, application retry logic, database settings, time-service design, settlement controls, non-production regions, vendor materials and engineers for experiments. The consultant will not rewrite the booking platform, operate production, choose a replacement vendor, redesign commercial settlement rules or certify statutory accounts; inaccessible failure modes will remain limitations in the opinion.

Why this is external work

The database team selected and configured the pilot, while the application team assumes legacy transaction behaviour that was never formally documented. The vendor can explain advertised guarantees but cannot provide an independent conclusion about the client’s financial invariants. External distributed-systems judgment is needed to create evidence the risk panel can challenge without turning the review into a product referendum.

What you will own

• Catalogue settlement transactions, invariants, isolation needs, ordering dependencies, retry behaviours, reconciliation controls and irreversible external effects.
• Translate database guarantees for consensus, leases, timestamps, replication and locality into observable application outcomes under specific faults.
• Design repeatable experiments for partitions, clock skew, leader loss, delayed replicas, ambiguous commits, duplicate clients and region isolation.
• Reconstruct histories from client, database and control evidence, identifying linearisation assumptions, unknown outcomes and compensating entries.
• Compare the tested behaviour with financial-control tolerances, recovery objectives, operational skills and cutover rollback constraints.
• Specify code, configuration, procedure and monitoring conditions that must close before any production data movement begins.
• Issue the evidence dossier and reasoned proceed, constrain or reject recommendation with an executable decision checklist.

Candidate qualifications

• Designed or independently assured distributed transactional databases under partitions, lease changes, clock faults and multi-region replication.
• Converted business or financial invariants into histories and fault tests rather than relying on availability and latency benchmarks.
• Diagnosed ambiguous commits, stale reads, duplicate retries or reordered effects whose business impact was initially hidden by successful responses.
• Built reproducible failure experiments with versioned configurations, workloads, client behaviour and evidence capture.
• Challenged database-vendor semantics using observed results while distinguishing configuration defects from product properties.
• Authored an independent migration or cutover opinion accepted by architecture, operations and financial-control stakeholders.

Non-negotiables

• Can conduct both Warsaw fault laboratories and the settlement-control workshop within the fixed twelve-week calendar.
• Will remain free of commercial ties to the incumbent database provider and any shortlisted replacement during the opinion.
• Brings direct multi-region transaction testing; data modelling, database administration or benchmark work alone is insufficient.
• Accepts repeatability, invariant reconciliation and a client-usable decision checklist as final acceptance requirements.

1. 49 words maximum. Describe one ambiguous commit pattern that an availability dashboard would miss but a settlement invariant would expose.
2. 49 words maximum. Which fault combination would you run first against a lease- or timestamp-dependent distributed database?
3. 49 words maximum. How would you distinguish a database guarantee failure from unsafe client retry behaviour in the captured history?