Your FastAPI Webhook Returned 200. Why Did the Order Run Twice?

Your webhook endpoint returned 200 OK.

The provider still delivered the event again.

That is not necessarily a provider bug. Most webhook systems promise at-least-once delivery, not exactly-once delivery. A timeout, a lost response, a process restart, or a manual replay can send the same logical event more than once.

The dangerous implementation looks harmless:

@app.post("/webhooks/provider")
async def receive(event: dict):
    create_order(event)
    return {"ok": True}

If the same event arrives twice, create_order() runs twice. Replace "order" with invoice, email, provisioning job, CRM record, or AI-agent action and the failure becomes expensive quickly.

Here is the small pattern I use to make the HTTP boundary safer.

1. Verify the signature against the raw body

Do not parse and re-serialize the payload before signature verification. Whitespace, key order, and encoding can change the byte sequence.

import hashlib
import hmac
def valid_signature(body: bytes, supplied: str, secret: bytes) -> bool:
    expected = hmac.new(secret, body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, supplied)

Use hmac.compare_digest() rather than a normal string comparison.

2. Treat the provider event ID as a database key

An in-memory set is not enough. It disappears on restart and does not coordinate multiple workers.

Create a durable inbox table with a unique event ID:

CREATE TABLE webhook_inbox (
  event_id TEXT PRIMARY KEY,
  event_type TEXT NOT NULL,
  payload TEXT NOT NULL,
  state TEXT NOT NULL DEFAULT 'accepted',
  attempts INTEGER NOT NULL DEFAULT 0,
  last_error TEXT,
  accepted_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);

The uniqueness constraint is the important part. Two concurrent requests can both pass a separate "does this ID exist?" query. Let the database perform the reservation atomically.

For a minimal SQLite example:

cursor = db.execute(
    """
    INSERT OR IGNORE INTO webhook_inbox(event_id, event_type, payload)
    VALUES (?, ?, ?)
    """,
    (event_id, event_type, body.decode()),
)
db.commit()
duplicate = cursor.rowcount == 0

In PostgreSQL, use INSERT ... ON CONFLICT DO NOTHING.

3. Acknowledge durable acceptance, not completed business work

Keep the HTTP path short:

1. read the raw body;
2. verify the signature;
3. validate the minimum envelope;
4. persist the inbox record;
5. return 202 Accepted.

Do not call three downstream APIs before responding. Slow processing increases the chance that the provider times out and retries.

The endpoint can look like this:

from fastapi import FastAPI, Header, HTTPException, Request, status
app = FastAPI()
@app.post("/webhooks/provider", status_code=status.HTTP_202_ACCEPTED)
async def receive_webhook(
    request: Request,
    x_signature: str = Header(default=""),
):
    body = await request.body()
    if not valid_signature(body, x_signature, secret):
        raise HTTPException(status_code=401, detail="Invalid signature")
    event = json.loads(body)
    cursor = db.execute(
        "INSERT OR IGNORE INTO webhook_inbox(event_id,event_type,payload) VALUES(?,?,?)",
        (str(event["id"]), str(event["type"]), body.decode()),
    )
    db.commit()
    return {
        "accepted": True,
        "duplicate": cursor.rowcount == 0,
    }

The worker processes accepted rows separately. A crash after the database commit does not lose the event.

4. Test the duplicate directly

Do not wait for production retries to discover whether idempotency works.

def test_duplicate_delivery_has_one_durable_acceptance():
    event = {"id": "evt_duplicate_1", "type": "order.paid"}
    first = signed_post(event)
    second = signed_post(event)
    assert first.status_code == 202
    assert first.json()["duplicate"] is False
    assert second.status_code == 202
    assert second.json()["duplicate"] is True

Also test:

• an invalid signature;
• two concurrent deliveries of the same ID;
• a worker crash after inbox persistence;
• a downstream timeout;
• events delivered in reverse order;
• a stale event trying to overwrite newer state.

5. Duplicate delivery is not the only problem

Events can arrive out of order. A delayed subscription.updated event may arrive after subscription.cancelled.

Store the provider timestamp or sequence number and define allowed state transitions. An older event should become a no-op rather than silently restoring stale state.

Retries also need a terminal state. Record attempt count, last error, and the next retry time. After the configured limit, move the event to a dead-letter state that can be inspected and replayed intentionally.

The practical rule

Your webhook should create one durable inbox record, not perform one irreversible business action inside the HTTP request.

That single design choice makes duplicates, crashes, retries, and debugging much easier to reason about.

I published the complete runnable FastAPI receiver, tests, production checklist, and reusable review prompt here:

https://codez.win/guides/fastapi-webhook-idempotency?utm_source=devto&utm_medium=article&utm_campaign=webhook_20260723

What is the most expensive duplicate webhook failure you have seen: double billing, duplicate emails, or repeated provisioning?