Webhooks are the backbone of modern event-driven architecture, connecting disparate systems and automating workflows in real-time. But what happens when you need to dispatch millions of automated event notifications daily to third-party endpoints?

Designing a highly available webhook delivery engine—much like the one powering GitHub—requires strict safeguards. You have to guarantee that failing, slow, or misconfigured external websites do not degrade your system’s performance or consume the resource pool of your healthy subscribers.

Here is a deep dive into the architecture, routing strategies, and resiliency patterns required to build a world-class webhook delivery system.

The Trap of Synchronous Delivery

When an event occurs, it is tempting to use a simple synchronous HTTP client loop to deliver the webhook immediately. However, this is a dangerous anti-pattern at scale.

Synchronous HTTP loops tie up application server threads while waiting for external networks to respond. Because third-party servers are inherently non-deterministic, they can be extremely slow, drop connections, or completely hang up. If your core services block while waiting for a slow website, your entire application thread pool will quickly starve, resulting in a system-wide crash.

To survive this, you must adopt an asynchronous event-driven model using a message broker to decouple event generation from outbound delivery.

Core Architecture & Decoupled Components

To build this asynchronous model, the system must be broken down into distinct, decoupled components:

Segregating Traffic with Tiered Queue Routing

How do you maintain high system throughput when external endpoints inevitably fail? The answer is Tiered Queue Routing.

• When an event triggers, it lands in a high-speed Fast-Track Queue.
• Dedicated workers attempt immediate delivery.
• If the endpoint returns a 200 OK success code, the task is complete.
• If the endpoint times out or returns a 5xx error, the worker immediately routes that subscriber’s subsequent tasks into a separate Low-Priority Retry Queue.

This guarantees that failing consumers only clog the slow retry lane, leaving the fast-track lane completely clear for healthy subscribers.

Preventing Multi-Tenant Exhaustion

Imagine a massive enterprise endpoint goes down completely. If the system continuously retries millions of their delivery logs, the retry queue overflows and your worker memory is exhausted. This is known as a Multi-Tenant Exhaustion vector.

To isolate this massive load and protect other users, you must implement two critical rules:

1. Lightweight Pointers: Message broker queues should never store heavy raw payload bytes. Instead, store lightweight text references containing only an event_id and a tenant_id. The heavy payload body is saved just once in a highly durable database.
2. Tenant-Based Rate Limiting: Enforce a strict concurrency cap. Once a tenant’s outbound error count spikes, their tasks are globally throttled at the broker level. This prevents a single broken domain from hijacking the worker capacity of clean tenants.

The Automated Circuit Breaker

What if an endpoint is permanently broken or returns 5xx errors for days? Retrying is a waste of computing power.

By maintaining an endpoint state tracker inside a distributed Redis Cache, you can implement an automated Circuit Breaker pattern. Every time a delivery attempt fails, the worker increments a failure counter. If the continuous failure count crosses a strict threshold (e.g., 20 consecutive drops or a 100% error rate over a 5-minute window), the Circuit Breaker trips and enters an OPEN state.

Once open, the Router Engine checks Redis and immediately short-circuits all new incoming events for that subscriber. These events are dropped instantly or routed directly to a suspension log without ever creating worker threads or executing useless network requests.

Securing the Payload & Guaranteeing Idempotency

Because network drops can happen mid-response, the delivery engine might send a payload successfully but fail to receive the 200 OK confirmation. Since the engine must treat connection drops as failures and reschedule the task, duplicate deliveries will happen.

To solve this, the burden of idempotency shifts to the consumer, aided by strict cryptographic metadata:

• Unique Execution IDs: Inject an immutable, unique header into every single delivery attempt, such as an X-GitHub-Delivery UUID. Consumers can check this UUID against their own deduplication layer to avoid executing the same event twice.
• Cryptographic Signatures: To prevent spoofing, every payload must be signed using a secret key shared between the platform and the user. The delivery engine calculates a keyed-hash message authentication code (HMAC-SHA256) and passes it via custom HTTP headers.
• Timestamp Verification: Include an X-Webhook-Timestamp header. The receiving server recomputes the hash and validates the timestamp within a tight window (e.g., 5 minutes) to guarantee the request is both genuine and fresh.

Handling the Dead Letter Queue (DLQ)

Eventually, some webhooks will exhaust their maximum retry policy (e.g., 5 attempts spread over 24 hours via exponential backoff). These are permanently evicted from the slow retry queue and pushed into the Dead Letter Queue (DLQ).

Tasks inside the DLQ are never automatically retried. Instead, they are persisted into a low-cost document store with an expiration policy (TTL) of 30 days. By exposing a public REST API and dashboard panel, users can inspect their exact failure stack traces, fix their server code, and trigger a manual retry that injects the payload right back into the Fast-Track Queue for immediate execution