When you are processing ad clicks at a global scale, you aren’t just counting numbers—you are handling money. A distributed ad click tracking system, like the one powering Google Ads, must process billions of events per day while guaranteeing absolute financial accuracy.

At a scale of 8.64 billion clicks a day, the system faces an average ingestion rate of 100,000 write operations per second (QPS), with traffic surging up to 300,000 QPS during peak events. You cannot afford to drop a click, but more importantly, you cannot affood to count a single click twice.

Here is a deep dive into how a high-throughput, exactly-once processing pipeline is engineered to handle extreme scale and intermittent network chaos.

The Trap of Naive Counters

If you try to track ad clicks using standard database atomic counters (like an SQL UPDATE or a standard Redis INCR), your system will implode.

When thousands of concurrent clicks hit the exact same advertiser campaign ID, it creates severe database lock contention. Furthermore, if a processing worker crashes midway through a job, restarts, and retries the batch, a naive insert script will double-count the same clicks. This leads to over-billing and massive financial losses for customers.

Event Time vs. Processing Time

To maintain financial accuracy, the architecture must strictly differentiate between two concepts:

• Event Time: The exact millisecond the user physically clicked the ad on their browser or phone.
• Processing Time: The time when the server engine actually runs the calculation code for that event.

Imagine a user clicks an ad while riding a subway at 11:59 PM, but their phone loses signal and only transmits the data at 12:05 AM. If the system logs the click using Processing Time, the revenue gets recorded on the wrong day’s financial report, causing a massive bookkeeping discrepancy.

Core Architecture: The Streaming Pipeline

To handle 300,000 QPS seamlessly, the system relies on an asynchronous, stream-based architecture:

Solving Row-Locking with “Budget Leasing”

If every incoming click immediately attempts to read and update an advertiser’s budget row in a central SQL database, the lock contention will crash the database.

To prevent this, the pipeline uses a Budget Leasing (or Token Bucket) pattern. The stream engine does not check the main database for every click. Instead, it pulls a small chunk of the budget—a “lease” of $10, for example—into local fast memory (RAM).

The system deducts the click costs locally in RAM at lightning speed. When the $10 lease runs out, the server requests another chunk from the main database. If the main database reports the total budget is gone, the server instantly stops accepting clicks for that ad, protecting the advertiser from overspending without slowing down the ingestion pipeline.

Handling Late Data and Network Drops (Watermarking)

How do you handle a click that arrives 30 minutes late without causing infinite processing delays?

The system uses a Watermarking mechanism combined with Allowed Lateness in the stream engine. A watermark is a monotonic clock reflecting how far behind the stream engine is relative to the event time.

By defining a Tumbling Window of 1 minute with an allowed lateness of 30 minutes, any delayed click arriving within that 30-minute grace period will trigger an incremental update to the dashboard. Clicks arriving after the allowed lateness are safely routed to a Dead Letter Queue (DLQ) for separate manual batch adjustments, protecting the real-time engine’s performance.

Defense Mechanisms: Fat Fingers and Bot Attacks

The system must defensively protect the advertiser’s budget from both accidents and malice:

• The “Fat-Finger” Filter: If a user accidentally double-taps an ad, the advertiser shouldn’t pay twice. The system uses a Sliding Window Deduplication Filter directly inside the Stream Processing Engine’s memory. By tracking the user_id and ad_id inside a short 5-second window, any exact matches are flagged as accidental duplicates, dropped from the billing pipeline, but preserved in raw logs for basic analytics.
• Click-Fraud Bot Attacks: A bot attack firing 10,000 clicks per second creates a Hot Key problem that can lock the database. To survive this, the system applies early-stage Rate-Limiting right inside the stream engine memory. If a specific user token or IP exceeds a pre-configured click threshold within 5 seconds, the worker flags the events as fraud, short-circuits the pipeline, and strips them from the financial billing queue entirely.

Ensuring Absolute Financial Accuracy

Even with exactly-once processing semantics, real-time streaming pipelines prioritize speed, which can occasionally cause minor data drift under severe network partitions.

To enforce absolute billing correctness before an invoice is ever generated, the architecture relies on a nightly Reconciliation Batch pipeline. Every midnight, a heavy batch job (via MapReduce or BigQuery) scans the raw, immutable event log files from cold storage, aggregates the clicks globally by event time, and reconciles them against the serving metrics table to detect and correct any financial variance automatically.