Your API Retried the Request. Your Customer Got Charged Twice.

A customer clicks the Pay button, the server sends the request to a payment provider, and the charge succeeds. The trouble begins when the response never reaches the browser because the connection drops. The frontend sees a timeout and retries the same request, while the backend treats it as a brand-new payment. One customer action has now created two valid charges.

This kind of failure is easy to miss because every part of the system appears to behave correctly on its own. The browser retries because it never received a response, the API processes a valid POST request, and the payment provider accepts the instruction it was given. The defect sits between those systems, where no component has enough context to know that the second request is a repeat. The same pattern can create duplicate orders, emails, subscriptions, shipments, support tickets, or background jobs.

The common advice is to retry failed requests, but that advice is incomplete. A retry is safe only when the server can distinguish a repeated business operation from a new one. For payment APIs, that usually means introducing an idempotency key, storing the original result, and protecting the write path against race conditions. The rest of this article walks through that flow using Node.js, Express, and PostgreSQL.

The bug starts with a normal-looking endpoint

Consider a small Express endpoint that creates a payment by calling an external provider. The code receives the request body, sends the amount and customer data to the provider, and returns the resulting payment object. Nothing about this route looks obviously unsafe during a basic code review, and the happy-path test is likely to pass without trouble. The risk appears only when the request succeeds remotely but the response is lost before the client receives it.

`app.post("/payments", async (req, res, next) => {
try {
const payment = await paymentProvider.charge({
customerId: req.body.customerId,
amount: req.body.amount,
currency: req.body.currency
});

res.status(201).json(payment);

} catch (error) {
next(error);
}
});`
From the client’s point of view, a timeout does not reveal where the failure happened. The request may have died before reaching the server, failed during processing, completed successfully but lost its response, or continued running after the browser gave up waiting. Retrying is often the only sensible option, but the server currently has no stable identity for the underlying payment. It sees two HTTP requests instead of one payment intent with two delivery attempts.

That distinction matters because transport requests and business operations are not the same thing. A customer may submit one payment while the network produces several attempts to deliver it. The API needs a way to connect those attempts to the same business action. That is what the idempotency key provides.

What an idempotency key actually represents

An idempotency key is a unique value generated for one logical operation, such as one checkout payment. The client creates the key before sending the first request and reuses it for any retry caused by a timeout or temporary failure. A completely new payment gets a completely new key. The key belongs to the business action, not to a single network call.

`const idempotencyKey = crypto.randomUUID();

await fetch("/payments", {
method: "POST",
headers: {
"Content-Type": "application/json",
"Idempotency-Key": idempotencyKey
},
body: JSON.stringify({
customerId: "cus_4821",
amount: 4999,
currency: "USD"
})
});`
Once the server receives the key, it can apply a clear set of rules. A new key starts a new operation, the same key with the same payload returns the earlier result, and the same key with different payment details is rejected. Those rules give the API enough context to decide whether a repeated request is a retry or an attempt to reuse an old identity for a different payment. They also make the client’s retry behavior predictable.

A key by itself is not enough, though. The server must remember what happened during the first request, including whether the operation is still running, whether it completed, and what response was returned. Without that stored state, the API can block a second charge but still leave the client without a useful answer.

Store the operation state and original response

A practical idempotency table should hold more than a list of used keys. It should store a fingerprint of the request, the current state of the operation, the HTTP status returned to the caller, and the response body that can be replayed later. A timestamp and expiry date are also useful because most systems do not need to retain every key forever. PostgreSQL is a good fit because a primary key can enforce uniqueness under concurrent traffic.

CREATE TABLE idempotency_records (
idempotency_key VARCHAR(255) PRIMARY KEY,
request_hash VARCHAR(64) NOT NULL,
status VARCHAR(20) NOT NULL,
response_status INTEGER,
response_body JSONB,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
expires_at TIMESTAMPTZ NOT NULL
);
The status field should describe the life of the operation in terms the rest of the system can understand. A record marked processing means the original request still owns the work, while completed means the API can safely replay the stored response. A permanent business failure, such as a declined card, can be stored separately from an uncertain technical failure. That distinction becomes important when the payment provider may have completed the charge before the application lost contact.

Replaying the original response is useful because the retrying client receives the same outcome as the first request would have returned. The API does not need to invent a new message or ask the client to guess what happened. This also keeps client logic simpler because a successful retry can look like a successful first attempt. The duplicate request becomes boring, which is exactly what a payment retry should be.

Why a simple lookup is still unsafe

The first version of this logic often checks for an existing key, creates the payment when no record is found, and then saves the result. That looks reasonable and works when requests arrive one after another. It fails when two copies of the request arrive at nearly the same moment. Both can check the database before either has created the record, so both believe they own the payment.

`const existing = await findIdempotencyRecord(key);

if (existing) {
return res
.status(existing.responseStatus)
.json(existing.responseBody);
}

const payment = await createPayment(req.body);
await saveIdempotencyRecord(key, payment);

return res.status(201).json(payment);`
The race unfolds quickly. Request A checks for the key and finds nothing, then Request B performs the same check and also finds nothing. Both call the provider, both create a charge, and only after that do they try to insert the same key into PostgreSQL. A unique constraint may reject one database insert, but it cannot undo the external payment that already happened.

The key must be reserved before the side effect begins, and that reservation must be atomic. In other words, only one request can successfully claim the operation. Every competing request must discover that someone else already owns it before any charge is created. PostgreSQL’s ON CONFLICT behavior makes this straightforward.

INSERT INTO idempotency_records (
idempotency_key,
request_hash,
status,
expires_at
)
VALUES ($1, $2, 'processing', NOW() + INTERVAL '24 hours')
ON CONFLICT (idempotency_key) DO NOTHING
RETURNING idempotency_key;
Only one request receives a returned row. That request owns the operation and may continue to the provider call. Every other request receives no row and must read the existing record to decide whether to replay the response, reject a mismatched payload, or report that the original request is still running. The database constraint becomes the final authority on ownership.

Prevent one key from changing its meaning

A client must not be allowed to send one amount with a key and later reuse that key for a different amount. Without a payload check, the server may return the earlier response for the wrong request, which creates confusing and potentially dangerous behavior. The fix is to store a stable fingerprint of the fields that define the payment. When the same key returns, the incoming fingerprint must match the stored value.

`import crypto from "node:crypto";

function createRequestHash(body) {
const normalizedPayload = JSON.stringify({
customerId: body.customerId,
amount: body.amount,
currency: body.currency
});

return crypto
.createHash("sha256")
.update(normalizedPayload)
.digest("hex");
}`
The payload must be normalized before hashing. Hashing an arbitrary object can produce different values when field order changes, optional values are omitted, or numeric formats differ even though the business request is equivalent. A fixed structure with known fields avoids that problem. The goal is not to fingerprint every byte of the HTTP body but to capture the fields that define the operation’s identity.

When the stored and incoming fingerprints do not match, the API should reject the request rather than replaying an unrelated result. A 422 Unprocessable Entity response works well because the request is structurally valid but conflicts with the meaning already assigned to the key. The error message should be direct enough for a developer to understand that the client reused the key incorrectly.

if (record.request_hash !== requestHash) {
return res.status(422).json({
error: "This idempotency key was used with a different request."
});
}

Putting the route together

The following route combines key validation, request hashing, atomic reservation, response replay, and the payment call. It is not a complete billing platform, but it shows the pieces that must work together. The code also passes the same idempotency key to the provider, which matters because the provider owns the final payment side effect. Local protection alone cannot close every failure window.

`import express from "express";
import crypto from "node:crypto";
import { pool } from "./database.js";
import { chargeCustomer } from "./payment-provider.js";

const app = express();
app.use(express.json());

function createRequestHash(body) {
return crypto
.createHash("sha256")
.update(
JSON.stringify({
customerId: body.customerId,
amount: body.amount,
currency: body.currency
})
)
.digest("hex");
}

app.post("/payments", async (req, res, next) => {
const key = req.get("Idempotency-Key");

if (!key) {
return res.status(400).json({
error: "Idempotency-Key header is required."
});
}

const requestHash = createRequestHash(req.body);
const client = await pool.connect();

try {
const reservation = await client.query(
INSERT INTO idempotency_records (
idempotency_key,
request_hash,
status,
expires_at
)
VALUES ($1, $2, 'processing', NOW() + INTERVAL '24 hours')
ON CONFLICT (idempotency_key) DO NOTHING
RETURNING idempotency_key,
[key, requestHash]
);

const ownsOperation = reservation.rowCount === 1;
if (!ownsOperation) {
  const existingResult = await client.query(
    `SELECT request_hash,
            status,
            response_status,
            response_body
     FROM idempotency_records
     WHERE idempotency_key = $1`,
    [key]
  );
  const existing = existingResult.rows[0];
  if (!existing) {
    return res.status(409).json({
      error: "The request state could not be determined."
    });
  }
  if (existing.request_hash !== requestHash) {
    return res.status(422).json({
      error: "The key was already used for another request."
    });
  }
  if (
    existing.status === "completed" ||
    existing.status === "rejected"
  ) {
    return res
      .status(existing.response_status)
      .json(existing.response_body);
  }
  return res.status(409).json({
    error: "A request with this key is still processing.",
    retryAfterSeconds: 2
  });
}
const payment = await chargeCustomer(
  {
    customerId: req.body.customerId,
    amount: req.body.amount,
    currency: req.body.currency
  },
  {
    idempotencyKey: key
  }
);
const responseBody = {
  id: payment.id,
  status: payment.status,
  amount: payment.amount,
  currency: payment.currency
};
await client.query(
  `UPDATE idempotency_records
   SET status = 'completed',
       response_status = $2,
       response_body = $3
   WHERE idempotency_key = $1`,
  [key, 201, responseBody]
);
return res.status(201).json(responseBody);

} catch (error) {
try {
await client.query(
UPDATE idempotency_records
SET status = 'recovery_required'
WHERE idempotency_key = $1
AND status = 'processing',
[key]
);
} catch (recordError) {
console.error("Could not update payment state", recordError);
}

next(error);

} finally {
client.release();
}
});`
This flow gives the API a clear owner for each operation and a clear answer for repeated requests. It also avoids marking uncertain failures as final, which would be dangerous when the provider may have accepted the charge. The recovery_required state signals that the system must reconcile the local record with the provider before deciding whether another attempt is safe. That recovery step is where many simplified examples stop too early.

The hardest failure sits between your database and the provider

Imagine that the provider creates the charge successfully, but the application crashes before PostgreSQL is updated. The idempotency record still says processing, while the customer’s card may already have been charged. A local database transaction cannot solve this because the provider and PostgreSQL do not share the same transaction boundary. Committing one does not guarantee that the other will commit.

The strongest defense is to send the same idempotency key to the payment provider when its API supports that feature. A retry can then use the same identity at both layers, allowing your API to prevent duplicate local work while the provider prevents a second external charge. When the provider does not support idempotent requests, store its operation reference as early as possible and build a recovery worker that checks the provider before issuing another charge.

This is why payment reliability should not be treated as a small middleware task. Retry rules, database constraints, provider behavior, logs, recovery jobs, and test coverage need to be considered as one system. A structured software development process helps teams account for those concerns during planning, coding, testing, release, and maintenance instead of adding them after a duplicate transaction reaches production.

Decide what a concurrent retry should receive

When a repeated request arrives while the first one is still running, the API needs a documented response policy. One option is to return 409 Conflict with a short retry delay, which keeps the behavior explicit and easy to reason about. Another is to return 202 Accepted with a status URL, which works well when payment creation is asynchronous. A third option is to hold the second connection briefly and poll the original record, though that approach needs strict time limits.

No single response pattern is correct for every API. The right choice depends on client behavior, expected payment duration, gateway timeouts, and whether the product already has a status endpoint. The key point is consistency. Clients should know whether to retry, poll, or wait without guessing from generic server errors.

Failure states also need careful treatment. A declined card is a final business result and can usually be replayed for the same key, while a provider timeout is uncertain because the charge may still have succeeded. Storing both as a generic failure would remove information the recovery process needs. A small state model such as processing, completed, rejected, and recovery_required is far safer.

Test the failure paths that happy-path tests miss

A single successful payment test proves almost nothing about retry safety. The most valuable tests send the same key twice, both sequentially and at the same time, then confirm that the provider receives only one charge request. Another test should reuse the key with a different amount and verify that the API rejects it. These cases prove that key ownership and request fingerprints work under normal retry conditions.

The harder tests simulate failure between systems. Complete the provider call, stop the process before the local record is updated, and then retry the request. The test should confirm that the system checks the provider or enters recovery instead of issuing another charge. You should also test a lost response after a successful database update, expired keys, queue redelivery, and records that remain in the processing state longer than expected.

These scenarios require more than isolated unit tests because the risk lives across the API, database, provider client, and recovery path. Concurrency tests, controlled failure injection, and integration-level checks are much more useful here. For teams that need deeper coverage across transaction flows and race conditions, working with a dedicated quality engineering team can help validate the full behavior under realistic load and failure conditions.

Keep enough operational evidence to recover safely

Idempotency defects are much easier to investigate when the system records the right facts. Logs should include the key or a safe hash of it, the request fingerprint, the provider transaction ID, the current state, the number of replays, and the time spent processing. Avoid logging full payment details or sensitive customer data. The aim is to reconstruct the operation without creating a new security problem.

Alerts should watch for records that stay in processing beyond the normal payment duration. Those records may represent stopped processes, provider timeouts, missing updates, or blocked recovery jobs. A sudden increase in duplicate requests can also reveal aggressive client retries, unstable network paths, double-click defects, or repeated queue delivery. Idempotency data often becomes a useful signal for problems outside the payment route itself.

Expiry rules deserve the same care. Keys should live long enough to cover the full retry window for browsers, gateways, queues, support staff, and recovery workers. A public API may keep them for 24 hours, while a financial workflow may require a longer period for audits or disputes. Never delete an unresolved record simply because its original expiry time has passed.

Make retries boring

A retry looks like a network concern until it creates a second charge. The underlying mistake is treating every HTTP request as a new business instruction even though networks, browsers, queues, and proxies can repeat the same message. A dependable API expects that repetition and gives every logical operation a stable identity. Once that identity exists, the server can reserve ownership, compare payloads, replay results, and recover from uncertain states.

The complete pattern has several parts that need to work together. The client generates one key per payment intent, PostgreSQL reserves that key atomically, a request fingerprint prevents misuse, the provider receives the same key when possible, and recovery logic handles the gap between external and local state. Tests then prove that the flow remains safe under concurrency, timeouts, crashes, and repeated delivery.

The goal is not to stop clients from retrying because retries are often necessary. The goal is to make repeated delivery uneventful, predictable, and safe for both the customer and the system. When the same request arrives twice, the customer should still be charged once, and the API should know exactly why.