How to Test Webhooks Properly: A QA Guide for Real Product Risk

Webhooks look simple from the outside.

One system sends an event. Another system receives it. Something updates.

A payment succeeds, so the order is marked as paid. A subscription renews, so the account stays active. A dispute opens, so support is notified. A delivery status changes, so the user sees the latest state. A repository event fires, so a workflow runs.

On paper, that sounds clean.

In real products, webhooks are rarely that clean.

They can arrive twice. They can arrive late. They can arrive out of order. They can fail silently. They can be retried hours later. They can hit the wrong endpoint. They can be accepted with a 200 OK while the actual business logic never runs.

That is why webhook testing deserves more attention than it usually gets.

A webhook is not just an API request. It is a bridge between systems. When that bridge behaves badly, the product can show the wrong state, charge users incorrectly, provision access twice, miss important events, send duplicate notifications or leave internal records out of sync.

What a webhook actually does

A webhook is a way for one system to notify another system when something happens.

Instead of your product constantly asking a provider whether something has changed, the provider sends an HTTP request to your endpoint when an event occurs.

For example:

Payment provider: payment succeeded

Your system:
- mark invoice as paid
- send confirmation email
- unlock subscription
- update account balance

That is useful because it makes systems more real-time.

But it also creates risk because your product is now depending on an external event arriving correctly and being processed safely.

A normal API request is usually initiated by your system or your user. A webhook is initiated by another system. That means you do not fully control when it arrives, how often it arrives, whether it is delayed, whether it is retried or whether related events arrive in the order you expected.

That difference matters.

The first mistake: only testing that the webhook arrives

Many teams test webhooks too narrowly.

The test becomes: trigger the event, check that the endpoint receives the payload, return 200 OK and mark it as passed.

That is only the beginning.

The better QA question is:

What happens to the product state after the webhook is received?

The real risk is rarely that the endpoint cannot receive JSON. The real risk is what the application does with the event.

Does it update the correct record? Does it ignore duplicates? Does it verify the signature? Does it handle missing fields? Does it process only final states? Does it avoid duplicate emails? Does it keep a log of processed events? Does it recover if the webhook arrives before the local record exists?

Those are the questions that matter.

Real scenario: payment succeeds, but the product does not update

Imagine a SaaS product using a payment provider.

The user pays for a subscription. The payment provider sends a payment_succeeded webhook. Your product receives it and should activate the account.

The happy path is simple: the user pays, the provider confirms payment, the webhook arrives, the account is activated and the confirmation email is sent.

But now test the real world.

The webhook arrives while your server is under load. The endpoint takes too long to respond. The provider retries. The same event arrives again. Your system processes both deliveries.

Now the user has one payment, but your product may have duplicate confirmation emails, duplicate subscription records, duplicate account-credit entries, duplicate audit logs or duplicate fulfillment actions.

That is a classic webhook failure mode.

A stronger test is:

Send the same webhook twice.
Send it quickly.
Send it after a delay.
Send it while the first one is still processing.
Check that product state changes only once.

Duplicate webhooks are normal

A common misunderstanding is that duplicate webhooks are rare edge cases.

They are not.

Many providers use retry behaviour because losing an important event is worse than sending it more than once. If your endpoint does not respond correctly, responds too slowly, times out or returns a non-2xx status, the provider may retry.

Stripe tells teams to handle duplicate events and also notes that manually resending a failed event does not cancel automatic retry behaviour. Shopify documents retry behaviour for failed webhook calls and says that continued failures can lead to webhook subscription removal.

GitHub recommends using unique delivery identifiers and validating deliveries. Paddle also warns that webhook events can arrive out of sequence.

The lesson is simple:

If your webhook handler cannot safely receive the same event more than once,
it is not ready.

Duplicate financial impact is not theoretical

Not every duplicate transaction incident is caused by a webhook. Some are caused by payment processors, hardware failures, banking glitches, vendor processing issues or other system failures.

But they show why duplicate processing matters.

TIME reported that Bank of America refunded Apple Pay users after around 1,000 transactions were affected by duplicate charges. ABC reported duplicate transaction issues affecting Commonwealth Bank customers. CNA reported that 375 OCBC customers were charged twice for AXS payments due to a processing issue. WIRED reported an older Wal-Mart and First Data incident where more than 800,000 card transactions were double- or triple-billed.

Those examples should not all be labelled as webhook bugs.

But they show the consequence of the same class of product failure:

one real-world action
→ represented more than once in the system
→ customer sees duplicate financial impact
→ business must repair, refund and explain it

If your product processes external payment events, duplicate handling is not a minor technical detail. It is a trust safeguard.

The second mistake: trusting event order

Another common webhook mistake is assuming events arrive in the same order they happened.

That assumption is dangerous.

A provider may generate several related events such as payment_created, payment_processing, payment_succeeded, invoice_paid and subscription_updated.

A developer may assume they will arrive neatly in that order.

But real delivery does not always work like that. Network delays, queue timing, retries, provider behaviour and internal processing can change arrival order.

Paddle tells developers to use the event occurrence time rather than arrival time when reasoning about sequence because webhooks can be delivered out of order. Stripe also warns teams not to depend on event delivery order.

QA should test this because real systems do it.

Real scenario: subscription stuck in the wrong state

Imagine a subscription product.

The provider sends subscription_created, invoice_paid and subscription_active events. Your system expects that exact order.

Now imagine invoice_paid arrives first. The payment has succeeded, but your local subscription record has not been created yet.

A weak handler may fail like this:

invoice_paid received
→ subscription record not found
→ handler returns 500
→ provider retries
→ user paid, but account stays inactive

The user sees payment successful, but the account is still locked. The team sees failed webhook deliveries, inconsistent state and a support ticket.

A good QA test should simulate this situation.

The better question is not whether subscription activation works when events arrive perfectly. The better question is whether the system eventually reaches the correct state when events arrive late, twice or out of order.

The third mistake: processing before verifying

Webhook endpoints are public-facing URLs.

That means security matters.

A proper webhook handler should verify that the event actually came from the provider. Most providers support signatures or secrets for this reason.

The dangerous version looks like this:

Receive webhook payload
→ trust it
→ update user account
→ unlock paid feature

A safer version verifies the signature, validates the event type, checks the event ID, processes safely and stores the result.

This is not only a security issue. It is also a QA issue.

QA should test valid signatures, missing signatures, wrong signatures, old timestamps, modified payloads, unsupported event types and unexpected payload structures.

A webhook test that never checks invalid or untrusted events is incomplete.

The fourth mistake: doing too much before responding

Many webhook handlers do too much work before responding.

They receive the event, update several records, call external APIs, send emails, trigger fulfillment, update analytics and only then return 200 OK.

That creates risk.

If the handler takes too long, the provider may think delivery failed and retry. If the process crashes after doing the work but before returning success, the provider may send the same event again. If one downstream service is slow, the whole webhook delivery may become unstable.

A more reliable pattern is:

verify event
store event
respond quickly
process asynchronously
mark processing result
retry internally if needed

GitHub recommends responding quickly and doing heavy work asynchronously. Shopify also makes response timing important because slow or failed responses can lead to retries and eventually subscription removal.

From a QA perspective, the test should include slow processing, crashes after partial updates and worker failures after the event has already been accepted.

Provider delivered does not always mean product updated

A frustrating webhook bug is when the provider dashboard says delivery succeeded, but the product state is still wrong.

For example, the provider shows 200 OK, but the order is still unpaid, the subscription is still inactive or the notification was not sent.

This can happen for many reasons. The endpoint may return 200 before business logic runs. The event may be stored but the worker fails. The handler may ignore the event type. The local record may be missing. The update may fail. Or the product may update the wrong user or object.

That is why QA should not stop at provider delivery status.

Delivery success is not the same as business success.

A good webhook test checks both:

Was the event delivered?
Was it processed?
Was the correct product state updated?
Was the side effect created exactly once?
Can support see what happened?

The fifth mistake: no reconciliation path

Webhooks are useful, but they should not be the only way to know the truth.

If a webhook is missed, failed, removed, ignored or processed incorrectly, the product needs a way to recover.

For payment and subscription products, this often means fetching the current state from the provider and reconciling local records.

A simple recovery pattern looks like this:

Webhook says payment succeeded
→ update local state

If webhook is missed
→ scheduled job checks provider API
→ local state is corrected

This is especially important for transitional states such as payment pending, invoice awaiting confirmation, subscription activating, withdrawal processing, delivery in progress or identity verification pending.

If those states depend only on a webhook, users can get stuck.

Recent research on enterprise messaging pipelines describes the same general problem: webhook callbacks can fail because of network issues, endpoint unavailability or provider retry exhaustion, leaving records stuck in intermediate states. A fallback polling path can reconcile those records later.

What proper webhook testing should cover

Webhook testing should cover more than one successful delivery.

A practical QA scope should include signature validation, duplicate delivery, out-of-order delivery, slow response behaviour, unsupported event types, missing local records, side effects, observability and replay safety.

For duplicate delivery, send the same event ID and payload more than once, both immediately and after a delay. The event should be processed once, and duplicate deliveries should not create duplicate side effects.

For out-of-order delivery, send related events in the wrong order. The system should handle the event safely, wait for missing context, fetch the latest provider state or eventually converge to the correct state.

For side effects, check email, notifications, account access, order status, subscription status, balance updates, audit logs, admin views and external fulfillment.

The goal is not only to see a 200 OK. The goal is to prove the product state remains correct.

What to include in a webhook bug report

A good webhook bug report should not only say “Webhook did not work.” That is too vague.

A useful report includes the environment, provider, event type, event ID, delivery ID, object ID, provider timestamp, local receive timestamp, relevant payload fields, steps, expected result, actual result, evidence and user impact.

For example:

Event type: invoice_paid
Event ID: evt_123
Object: subscription_456

Expected:
Subscription should activate once.

Actual:
Two subscription records were created and two emails were sent.

That kind of report helps developers faster because it points to the actual failure mode instead of only describing the visible symptom.

How teams can build safer webhook systems

Good webhook handling usually has several layers.

A safer design often includes signature verification, event ID storage, a deduplication table, idempotent business logic, quick acknowledgement, background processing, retry-safe workers, event status logs, dead-letter queues, manual replay tools, provider-state reconciliation, monitoring and alerts.

The exact architecture depends on the product.

But the principle is the same:

Do not treat a webhook as a one-time perfect message.
Treat it as an external event that may be late, repeated, missing or out of order.

That mindset changes how the product is built and tested.

Where Laidoner Solutions helps

Webhook bugs are a good example of why QA should look beyond the visible UI.

A user may see a simple message like “Payment successful.” But behind that message, several systems may need to agree: the payment provider, backend API, database, subscription service, email service, admin panel, audit logs, frontend state and support tools.

Laidoner Solutions helps software teams test these kinds of product risks through practical QA, API testing, regression checks and product-quality review.

For webhook-driven products, this can include testing duplicate event handling, checking retry behaviour, validating webhook security handling, testing out-of-order events, checking payment and subscription state updates, checking notification side effects, reviewing admin and audit views, testing missed-event recovery and creating clear bug reports with provider and local evidence.

The goal is not only to check whether the webhook endpoint returns 200 OK.

The goal is to check whether the product stays correct when real systems behave imperfectly.

Final thoughts

Webhooks are easy to underestimate because they look small.

One endpoint. One payload. One response.

But in many products, webhooks control important state. Payments, subscriptions, orders, deliveries, account access, notifications and workflow automation may all depend on them.

That makes webhook testing important.

The useful question is not whether the webhook was received.

The better question is:

Did receiving this webhook move the product into the correct state,
exactly once,
even if the event was late, duplicated or out of order?

That is the difference between testing the integration and testing the risk.

And in financial or subscription products, that difference can be expensive.