·HOOX·2 min read

[The Mutex at the Door: Two-Phase Durable Object Idempotency]

Two-phase reserve/commit/release DO mutex—at-most-once gateway acceptance, fail-closed if missing, soft-fails release.

hooxdurable-objectsidempotencydeduplicationwebhooksreliabilitytrading-systemscloudflare

Sources: papers/sections/04-mechanisms.tex, threat-model appendix notes, docs/enduser/concepts/idempotency.mdx, gateway v0.13 mesh hardening.

Thesis

Webhook platforms retry. Networks drop acks. In trading, “please try again” is how double positions are born. HOOX’s answer is not a hopeful KV flag: it is a Durable Object two-phase mutex—single-threaded execution, SQLite-backed storage, reserve → commit / release under blockConcurrencyWhile.

Claim (precise): at-most-once acceptance at the gateway within TTL, with failed attempts free to retry after release.
Non-claim: formal exactly-once across Queues (at-least-once) and exchange APIs.

1. The double-fill story

Without dedup: fill succeeds, response lost, client retries, second fill.
With the DO: second attempt collides on a pending or committed key inside the TTL and is rejected (e.g. 409).
If the first attempt soft-fails (success: false even with HTTP 2xx), the gateway releases so the next try can reserve again.

2. Algorithm (two-phase)

Input:  key k, TTL τ
Output: proceed | duplicate | unavailable

# Phase 1 — before queue/service work
enter blockConcurrencyWhile
  e ← storage.get(k)
  if e is committed or pending and (now − e.storedAt) < τ then
    return duplicate
  storage.put(k, { status: pending, storedAt: now })
  schedule alarm at now + τ
  return proceed

# Phase 2a — true success (trade path OK)
commit(k) → status = committed   # retries stay blocked for TTL

# Phase 2b — soft-fail or hard error
release(k) → delete / free key   # retries may reserve again

Missing IDEMPOTENCY_STORE binding is fail-closed (503 / IDEMPOTENCY_UNAVAILABLE) — not “skip dedup and hope.”

Default TTL is short (minutes). Alarms reclaim storage so the object is not an infinite ledger of every trade ever attempted.

3. Key granularity — deliberate trade-off

Preference order:

  1. Client idempotencyKey / Idempotency-Key header (cross-minute uniqueness)
  2. Auto fingerprint:
trade:{exchange}:{symbol}:{action}:{quantity}:{live|test}:{minuteBucket}

The per-minute bucket means intentional same-size trades later are not blocked for the full TTL. Live and test modes never share a key. Design around client keys if you need burst identical size-ins; do not design around accidental double webhooks.

4. Fail-closed vs fail-open (v0.13 posture)

Concern Posture Rationale
Authentication Fail closed Forged trade is catastrophic
Idempotency DO unavailable Fail closed Trading without the mutex is worse than a brief refuse
Soft-fail after reserve Release Operator/retry must be able to re-enter

Documented in the threat model and gateway tests. Production rule:

Authentication fails closed; missing idempotency store fails closed; soft-fails release so retries work.

5. Why DO, not KV

KV is brilliant for config and rate windows—eventual consistency. Wrong tool for “exactly one of two concurrent POSTs may proceed.” Durable Objects give strong consistency for one key space, no get/put interleaving under concurrency, and alarms without a janitor worker.

Sister objects: RateLimiterStore for atomic multi-isolate rate limits; ExchangeConnectionManager amortizes WebSockets and is not the idempotency or order path (essay 11).

Previous: Seven-stage signal · Next: Service Bindings fabric

All postsHome