Digital Media Store Design: Idempotency Is Non‑Negotiable in Purchases

Why idempotency matters for purchases

In a Digital Media Store, the purchase endpoint must be idempotent. Networks fail, clients retry, and gateways time out—so the same request can hit your backend multiple times. If you don't design for idempotency, you'll risk double‑charging users or creating duplicate PURCHASE records. That harms revenue, user trust, and data integrity.

The rule (simple and non-negotiable)

Treat POST /purchase as a transaction keyed by an Idempotency-Key. Store the key along with a status (PENDING / SUCCESS / FAILED) and the payment transaction_id or error details. On retry, return the original result instead of reprocessing the payment.

This single pattern prevents duplicate charges, simplifies retry logic, and makes behaviors deterministic.

Recommended implementation pattern

1. Client generates an Idempotency-Key (e.g., UUID v4) and sends it in a header: Idempotency-Key: <uuid>.
2. Server receives the request and looks up the key (scoped to the user/account or global depending on your requirements).
3. If the key is new, insert a record with status = PENDING and start processing the payment.
4. If the key exists and status = PENDING, return the existing pending response or wait/stream updates.
5. If the key exists and status = SUCCESS or FAILED, return the stored result (success payload or error) without reprocessing.

Example idempotency table schema (conceptual)

idempotency_keys
-----------------
id               UUID PRIMARY KEY
user_id          UUID         -- optional, scope the key
idempotency_key  TEXT UNIQUE  -- or use (user_id, idempotency_key)
status           TEXT         -- PENDING, SUCCESS, FAILED
created_at       TIMESTAMP
updated_at       TIMESTAMP
payment_txn_id   TEXT NULL    -- payment gateway transaction identifier
response_body    JSON NULL    -- serialized response to return for retries
error_code       TEXT NULL
expiry_at        TIMESTAMP    -- TTL for cleanup

Guidelines:

• Enforce a uniqueness constraint on (user_id, idempotency_key) to avoid races where two inserts try to create the same key.
• Write the initial PENDING row within a transaction or via an atomic upsert so only one worker proceeds to process the payment.

Workflow (detailed)

• Client: POST /purchase with body and header Idempotency-Key: abc.
• Server: BEGIN TRANSACTION
  • Try to insert idempotency row with status = PENDING. If insert fails because key exists, SELECT the row.
  • If row.status == PENDING: return a 202/200 with the pending state or wait depending on your UX.
  • If row.status == SUCCESS or FAILED: return the stored response_body and status.
  • If this worker created the PENDING row: call the payment gateway.
    • On payment success: update row to SUCCESS, set payment_txn_id and response_body, commit.
    • On payment failure: update row to FAILED, set error_code and response_body, commit.
• Return the response saved in response_body for all retries.

Handling concurrency and races

• Use a unique constraint and an atomic insert/upsert so only one process will see itself as the owner of the PENDING row.
• If you need to avoid blocking clients, return a consistent response for PENDING and provide a mechanism to query status (e.g., GET /purchase/status?key=...).
• Alternatively, use SELECT ... FOR UPDATE on the idempotency row to serialize processing for that key.

What to store in response_body

Store the minimal canonical response that you return to the client on success or failure, including HTTP status code and body (e.g., receipt id, purchased items, errors). This lets retries receive exactly the same result.

Edge cases and operational concerns

• Long-running payments: mark PENDING and consider a reasonable timeout before marking FAILED. Use payment gateway webhooks to update final status asynchronously.
• Partial failures / timeouts: a client may timeout but the payment completes. When the client retries with the same key, return the SUCCESS stored result.
• Reconciliation: keep logs and reconcile with your payment provider using payment_txn_id to detect anything missed.
• Cleanup: TTL old idempotency rows (e.g., 30–90 days) with a background job to avoid unbounded growth.
• Security: scope keys to the authenticated user to prevent cross-account replay.

Client guidance

• Clients should generate a fresh Idempotency-Key per logical purchase attempt (UUIDs are fine).
• Retry the same key on communication failures; on a user-initiated new purchase, generate a new key.
• Do not reuse keys across different purchase intents or amounts.

Quick pseudocode

if not exists (select 1 from idempotency_keys where user_id = U and key = K):
    insert (K, user_id=U, status=PENDING)
    process_payment()
    if success:
        update idempotency_keys set status=SUCCESS, payment_txn_id=..., response_body=... where key=K
    else:
        update idempotency_keys set status=FAILED, response_body=... where key=K
    return response_body
else:
    row = select * from idempotency_keys where key=K
    return row.response_body

Testing and observability

• Test retries by forcing client or network failures and asserting no duplicate charges.
• Log idempotency key lifecycle transitions (PENDING -> SUCCESS/FAILED) and payment_txn_id.
• Monitor metrics: number of duplicate requests, rate of retries, time spent in PENDING.

TL;DR

Make POST /purchase idempotent using an Idempotency-Key. Store key + status + payment_txn_id + canonical response. On retries, return the saved result instead of reprocessing. This pattern protects revenue, preserves user trust, and keeps your data clean.

#SystemDesign #DistributedSystems #BackendEngineering