Why Did We Get Charged Twice? Designing Idempotency for Payments

This class of bug usually doesn't look like a bug when it first shows up. The typical pattern: customers report two charges, finance sees two captures of the same amount a few seconds apart, and the order system shows one order. Nobody clicked twice. The system charged twice on its own, and it will keep doing it whenever the same conditions come back.

I've spent 16 years building and running production software, most of it at Plondo Network. That includes direct-sales platforms used by 500K+ people since 2012, with 99.9% uptime on the systems that pay out commissions. Systems like that retry all the time, and every retry has to answer the same question safely: has this already happened?

How does a timeout turn into a second charge?

The sequence is almost always the same:

1. Your server sends a charge request to the processor.
2. The processor authorizes and captures the card.
3. The response never arrives. A load balancer times out, a mobile connection drops, or a deploy kills your worker.
4. Your code only sees a timeout. It can't tell whether the charge failed or went through, so it does what it was built to do and retries.
5. The processor treats the retry as a new request and captures again.

The retry logic isn't the mistake. The mistake is sending the retry with nothing that marks it as a repeat of the first request. Distributed systems will always retry. What matters is whether a repeated request carries an identity that lets something downstream say "already done, here is the original result."

Stripe recommends an idempotency key on all POST requests. The key should be unique, such as a UUID or a combination of customer ID and order ID, so a request can be retried safely after a network error. Stripe also suggests V4 UUIDs or another random string with enough entropy to avoid collisions.

The key has to identify the logical operation, not the HTTP attempt. Generate it once when the order or payout is created, save it, and reuse it on every retry. Code that generates a fresh UUID for each attempt has rebuilt the original bug with extra steps.

What does an idempotency key actually have to guarantee?

Stripe is the reference implementation. It saves the status code and body of the first request for a given key, whether that request succeeded or failed, and returns the same result for later requests with that key, 500 errors included. Your own API should promise its clients the same thing: one key, one outcome, replayed exactly.

Most tutorials skip the hard part, which is storage. The classic bug writes the idempotency record and the charge record as two separate, non-atomic steps. If the process dies between them, or two requests interleave, one request's check misses the other's write. The key and the charge have to be created in the same database transaction, and the database has to enforce uniqueness. Application code that checks first and inserts second can't.

A minimal PostgreSQL shape:

CREATE TABLE idempotency_keys (
  tenant_id      bigint      NOT NULL,
  idem_key       uuid        NOT NULL,
  request_hash   bytea       NOT NULL,          -- detects same key, different payload
  state          text        NOT NULL DEFAULT 'started',
  charge_id      bigint,
  response_code  int,
  response_body  jsonb,
  created_at     timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (tenant_id, idem_key)
);

-- Phase 1: claim the key and create the pending charge atomically
BEGIN;
INSERT INTO idempotency_keys (tenant_id, idem_key, request_hash)
VALUES ($1, $2, $3)
ON CONFLICT (tenant_id, idem_key) DO NOTHING
RETURNING idem_key;
-- 0 rows returned: key already claimed. Roll back, then replay the
-- stored response, or return 409 if state is still 'started'.
INSERT INTO charges (tenant_id, idem_key, amount_cents, status)
VALUES ($1, $2, $4, 'pending');
COMMIT;

The processor call can't go inside that transaction, and you wouldn't want it there. Holding a database transaction open across a network call to a third party is its own outage waiting to happen. So the flow has 3 phases:

• Phase 1 (one transaction): claim the key and insert the charge as pending.
• Phase 2: call the processor with a provider idempotency key derived from your key, so the processor dedupes too.
• Phase 3 (one transaction): record the processor's result, set the key's state to completed and store the response for replay.

If the process crashes between phase 1 and phase 3, a recovery job picks up started rows older than a threshold and repeats phase 2 with the same provider key. The processor replays its original result instead of charging again. Each layer of idempotency covers the gap the other one can't.

Compare request_hash on every replay. If a client reuses a key with a different amount or customer, reject the request with a clear error. Never quietly replay a response that belongs to a different request.

Is a unique constraint enough, or do you also need a distributed lock?

With a single primary database, the primary key on idempotency_keys does most of the work. The INSERT ... ON CONFLICT is atomic, so exactly one request wins. A lock helps when the check-then-act spans several services or datastores, or when you want to turn away in-flight duplicates quickly without hitting the database.

If you use Redis, take the lock in one atomic command. A naive SETNX followed by a separate EXPIRE is a race: a crash between the two commands leaves a lock that never expires.

SET lock:charge:{tenant}:{idem_key} {random_token} NX PX 30000
# release only if you still own it (compare-and-delete in a Lua script)

Know what the lock can't do. Martin Kleppmann's analysis shows that Redlock has no mechanism for generating fencing tokens. So even a correct implementation can't stop a race when a client pauses (a long GC pause, for example) or its packets are delayed past the lock's TTL. The process wakes up, still thinks it holds the lock, and keeps going.

That's why the robust pattern pairs a distributed lock with a database uniqueness constraint as a safety net. The lock handles the normal path, and the constraint catches edge cases such as two holders after a failover. The same guide says that for financial transactions, a single-node Redis lock's failover behavior is considered unacceptable, and it recommends Redlock, etcd or Consul when double execution is expensive. My view: the lock is an optimization and the constraint is the guarantee. If taking the lock out would let you double-charge, the design isn't finished.

Why aren't idempotency windows portable between processors?

Keys expire, and each provider sets its own expiry. Retry logic written against one processor can bring duplicates back when you add a second gateway or migrate.

Stripe's v1 and v2 windows are documented in its API v2 overview. The Adyen details, including the 7-day minimum and the explicit in-progress error, come from an analysis that compares the two providers' docs.

In practice, keep your own key table longer than any provider's window, and make sure your recovery job finishes well inside the shortest window you rely on. If a reconciliation job retries a stuck v1 charge 30 hours later, the processor sees a new charge.

Commission payouts have the same bug, with a worse audience

Card charges get most of the attention, but commission engines fail the same way. A webhook is redelivered, or a nightly batch reruns after a partial failure, and an agent gets paid twice. Asking a sales rep to give money back is a harder conversation than refunding a customer.

The fix is the same idea applied to a ledger:

• Derive the key from business identity, such as payout:{period}:{agent_id}:{plan_version}, not from a job run ID.
• Write payouts as append-only ledger entries with a unique constraint on that key. Correct mistakes with reversing entries, never with updates.
• Make the batch resumable, so a rerun skips entries that already exist and creates only the missing ones.
• Store the processor's transfer ID on each entry so reconciliation is a join, not a hunt.

A double payout is also an audit question: can you show exactly what was paid, why, and that each item was paid only once? An append-only ledger answers that by design.

Keep idempotency separate from card-security compliance, though. PCI DSS applies to any business that stores, processes or transmits cardholder data, regardless of size, but passing a PCI assessment tells you nothing about whether your retries are safe.

How do you test replay, timeout and concurrency before trusting it?

Idempotency nobody has tried to break is a guess. Before real money flows, I want each of these to pass in an environment pointed at the processor's sandbox:

• Replay: send the same key twice in a row. Expect one charge, and the second call returns an identical body and status.
• Payload mismatch: reuse a key with a different amount. Expect a rejection, not a replay.
• Timeout: use a proxy to drop the response after the processor has captured, and let the client retry. Expect one capture.
• Crash mid-flow: kill the worker between phase 1 and phase 3, then run recovery. Expect one capture and a completed key.
• Concurrency: fire 20 to 50 identical requests at once from separate connections. Expect exactly one winner, with the rest replayed or rejected as in progress.
• Lock expiry: pause the lock holder past the TTL (a debugger breakpoint works). Expect the unique constraint to reject the second writer.
• Window edge: retry after the provider's replay window has closed. Confirm your own table blocks it.
• Reconciliation: compare your ledger with the processor's records for the test run. Expect zero orphans in either direction.

Automate the concurrency and timeout cases in CI. Those are the ones that break when someone refactors the payment client later.

Where to start

1. If you've had a duplicate, reconstruct the exact sequence: which call timed out, what retried it, and which key (if any) went with the request.
2. Check whether keys are generated once per operation and saved, or regenerated on each attempt.
3. Confirm the key and the charge or payout record are created in one transaction behind a real unique constraint.
4. Send a stable provider idempotency key on every processor call, and check each provider's window.
5. Run the test list above against a sandbox before the next release.
6. Reconcile past periods against processor records to find other duplicates, and refund them before customers ask. In Louisiana, a customer stuck with an unresolved billing dispute can file a complaint with the Attorney General's Consumer Protection Section at 1-800-351-4889 under the state's Unfair Trade Practices and Consumer Protection Law. Get the refund out before it reaches that point.

If you'd like a second set of eyes on a payment or commission engine that has double-charged or double-paid, email hello@josephleporati.com.