Key Takeaways
- Synchronous webhook to runner systems do present some issues but we see that they are at risk for certain problems. The solution isn't to add more runners, instead what we do is we separate input functions from processing via the partitioned event bus.
- Partitioning effected on the basis of organization, type of repository, type and name of reference preserves arrangement of references while avoiding the creation of hot partitions for big repositories. Because there is a lack of needs of the execution process, the stateless receiver only needs to verify and publish.
- When queuing arises, the protocol takes a new form. As it happens, the messages can be delayed, restored, and reverted which implies that some verification must happen before the process of publishing, deduplications on the basis of delivery ID must occur at the moment of delivery and processing should also be checked once again.
- One must consider that there can be at least one delivery failure. In this case, the idempotent nature of keys allows for eliminating duplicates; the usage of the dead lamb queue allows for processing of bad messages without interruptions; the occurrence of the consumer lag allows for realization of the potential delivery backlogs.
Introduction
In a typical situation we see a report of a dependency update which in turn affects hundreds of repos in 10 minutes. The code may still do its job fine but what we see is all the repos at once putting out push and pull request events. In our case we have a sync design that is -- for auth, routing, and delivery also sync. Thus the CI receiver is hit by a lot more traffic then is usual, also with no buffer. Some requests will go over the delivery time out, some will get caught up in busy runners, and some will just disappear. Our CI receiver doesn\'t handle redelivery of broken webhooks so these lost requests may reappear in a few hours as pull requests which didn\'t get their status check.
CI traffic is atypical of the fact that we are the source of our own spikes. We see mass dependency updates, a company wide rebase or a nightly job at the same cron tick for each repo which produces a trend that is perfectly in sync across a wide range of sources. AI has made this issue bigger. We are going for a 3x in developer productivity through AI support and also our agents are opening pull requests in some cases across many repos at once. Also, industry research reports that as we adopt AI more into our work flow we see better delivery throughput [2].
This article discusses the workflow, which we created for the CI platform with the goal of achieving 99.99% availability. It also provides details about implementation features making it possible to achieve this level of availability and tells what mistakes we made while working on this project.
Why the Synchronous Webhook Design Breaks
In the synchronous design, the source-control system sends a signed HTTP POST for each push, pull-request update, or tag [3], and the receiver validates it, selects a pipeline, and dispatches to a runner in the same request. Its response time becomes the sum of every downstream call, while GitHub expects a response within ten seconds [4]. Being able to send and receive events is not the issue that we struggle with. Our platform can process almost ten million events in a week; and traffic flow is anything but balanced: peaks are around thirty times higher than the average and happen instantaneously. Kafka can deal with such loads easily. What fails, though, is the connection between receiving and responding to an event, which results in the following four responsibilities.
- Burst absorption. In the lack of a buffer the receiver has to at once deal with the peak as it happens, also auto scalers do not respond fast enough to 30x peaks.
- Partial failures as the new normal. When working at this scale, one component is always down, which creates a bottleneck.
- Ordering vs throughput. Events for a particular branch or pull request have to be processed in the right order to ensure that an old event does not overwrite a new one. Global ordering is not required and cannot be achieved.
- Untrusted and replay-worthy input. Queued events go through delay, redelivery, or replay which present a very different set of issues as compared to a verified request at the time of issue.
Adding runners does not resolve any of those problems.
Architecture and Design
Figure 1 shows the pipeline: A stateless receiver which does validation and publication of events, a partitioned Kafka topic, a consumer group scheduler which uses a database of branch and pull request state, an autoscaled execution layer, a rate limited status write back path to the GitHub API, a dead letter queue and a reconciliation job.
The receiver does three things: Accept requests, check their signatures, and report to Kafka. Also this does not have any other requirements in the request path which means it scales out without coordination and performs well within the platform’s timeout. We only acknowledge the webhook once Kafka has durably recorded the write; by returning success before that we actually reintroduce event loss.
The partition key is the most consequential design choice. We key every event as org/repo/refType/refName, where refType is branch or pr. Events with the same key are consumed in order, while different keys are processed in parallel [5][6]. Concurrently running which is what an event's record and the builds it triggers has gone into. We found out that how the queued builds behave is a per customer configuration. In that which comes to pull requests and feature branches teams tend to batch which means a new commit may leave out older queued builds or may in fact cancel them out of the queue also it allows for the possibility of the older builds to run at the same time. On the main branch we by default run a build for every commit as teams usually want a report for each and every release or audit they do.
The topic has a replication factor of 3 and holds events for a period of 60 days. Long term retention of events means that which have gone into dead letter queues may be replayed once a fix is in place, also in the time of which new consumer groups can catch up with the full history and deduplication window will equal the retention window.
The scheduler is a Kafka consumer group: each instance owns a subset of partitions and the group rebalances as instances join, leave, or fail [7]. Rebalance suspends usage on the partitions that have been transferred from one place to another. Although cooperative rebalance [8] and static membership [9] minimize this process, the automatic scaling agent remains very active, as it is demonstrated in the section of the study referring to experience gained.
Implementation
Verifying and publishing at the edge
GitHub webhooks carry an HMAC-SHA256 digest of the raw request body in a header [10]. The receiver recomputes it over the unparsed bytes and compares in constant time [11], because a non-constant-time comparison leaks timing information an attacker can use to forge a signature.
async function verifyAndPublish(rawBody, headers, secret) {
// Header format is "sha256=<hex digest of raw body>"
const expected = 'sha256=' + hmacSha256Hex(secret, rawBody);
if (!constantTimeEquals(expected, headers['x-hub-signature-256'])) {
return reject(401);
}
const event = normalize(parse(rawBody));
event.deliveryId = headers['x-github-delivery']; // dedup key for replays
// refType is "branch" or "pr"; ordering is preserved per key
const key = [event.org, event.repo, event.refType, event.refName].join('/');
await publish('ci.build.events', key, event); // wait for durable ack
return accept(202); // queued, not processed
}
The await on publish matters most: the producer waits for acknowledgment from the in-sync replicas and if Kafka is unavailable the receiver returns an error rather than accept an event it cannot store.
Committing consumer progress correctly
The offset is what Kafka uses for “to date all has been processed. Post processing commit provides at least once delivery [12]. Exactly once semantics break at the first external call [13] which is when the scheduler puts together a deterministic idempotent key for each processing job out of repo, commit SHA, pipeline, and trigger, also the execution layer which we put in to play for duplicate prevention.
Commit time goes up as we see unexpected order of events due to debounce (which is explained later). By default at the end of the process the Spring Kafka does a commit. But in our case the debounce listener goes off before the processing does which means that in the off chance of a crash during that delay the event may be lost. This can be fixed if we do the acknowledgment manually which in turn allows the code to do the commit only after we are really done with the processing.
Manual backs alone still allow out-of-order commits. If offset 100 is still waiting while 101 and 102 finish, committing 102 and then crashing loses event 100. Spring Kafka's asyncAcks prevents this by deferring each commit until every earlier offset has been acknowledged [14]. A simplified version:
factory.getContainerProperties().setAckMode(AckMode.MANUAL);
factory.getContainerProperties().setAsyncAcks(true);
@KafkaListener(topics = "ci.build.events")
void onEvent(BuildEvent event, Acknowledgment ack) {
if (event.isBranch() && !event.isDefaultBranch()) {
// Debounce off the listener thread; the partition keeps flowing
delayed.schedule(() -> {
resolve(event);
ack.acknowledge();
}, 2, TimeUnit.SECONDS);
} else {
handle(event);
ack.acknowledge();
}
}
Deduplicating branch and pull-request builds
A push to the head branch of a pull request generates an event for the branch and a separate event for the pull request, meaning that they can get delivered in any order. It would be wasteful to build two separate events for each pull request.
Upon getting a branch event for any branch that is not the default one the scheduler puts it on hold for a 2 second period and then checks its db for any present pull request which has the same branch name at its head. When it identifies a proper pull request it goes forward without the need to build out the branch, instead just the head commit SHA is updated, which is handled by the pull-request pipeline for the actual build. Also if the pull request’s status is not known at that point the scheduler does a slower but more accurate query to the GitHub API to determine if there is in fact an open pull request. This also helps to avoid duplicate builds, a thing that does not play out at all when using the default branch which in turn can be put forth as the most active branch in the monorepo.
In the resolution of out-of-order events, timestamps prove to be very useful at keeping events in chronological order and at the same time assuring present moment information is reliable. This system helps assign timestamps to each event transformation, depending on the timing of the event itself determined by the type of event, such as the branch event timestamp or timestamp of the last update in case of a PR event. In instances when an event falls under the out-of-order event category due to being older than the event state preserved so far, it is rejected. Delivery time cannot be utilized in this scenario, as in such cases the event while being delayed or undelivered may seem to occur after, while commit dates cannot be used either, as newly created commit timestamps may contradict the actual timestamp of zero commit.
Sizing the Partition Count
In the issue of out-of-order events timestamps play a key role in maintaining chronological accuracy of events at the same time they see to it that we receive reliable info in the present moment. This system helps assign timestamps to each event transformation, depending on the timing of the event itself determined by the type of event, such as the branch event timestamp or timestamp of the last update in case of a PR event.
In instances when an event falls under the out-of-order event category due to being older than the event state preserved so far, it is rejected. Delivery time cannot be utilized in this scenario, as in such cases the event while being delayed or undelivered may seem to occur after, while commit dates cannot be used either, as newly created commit timestamps may contradict the actual timestamp of zero commit.
- Baseline: divide observed peak load by 20 events/second to get the minimum partition count.
- Skew allowance (1.5x) for busy organizations and refs.
- Growth headroom (3x), matching the AI productivity target. Partitions cannot be removed, and adding them later changes the key-to-partition mapping.
- Round up to a power of two. We use 128 partitions, which gives roughly 2,500 events per second of scheduling capacity and allows up to 128 scheduler instances, though far fewer normally run.
Performance and Scalability
Partition out count, consumer parallelism, execution layer autoscale and tenant isolation. These have to be tuned in tandem. Four elements which determine if this architecture scales are partition count, consumer parallelism, execution layer autoscaling and tenant isolation.
- Partition count caps consumer parallelism, because a consumer group can have no more active members than the topic has partitions [6]. Choose it with headroom. By adding partitions later on, keys will be remapped and the ordering per-reference will be briefly disrupted, plus the number of partitions cannot be minimized. Consumer batching which processes multiple events per poll instead of one at a time increases the throughput per consumer but at the same time causes higher latency per event in lightly loaded conditions. This should be made a variable instead of a hard code.
- Autoscaling should follow leading signals. Consumers should be scaled based on the lag of the consumers and the runners based on the depth of the pending builds queue, instead of measured CPU or memory usage.
- At that point, the resource metrics will show saturation, with an upstream request backlog already in place.
- In terms of the isolation of tenants, it is important to note that hash partitioning involves a lot of tenants in each partition meaning that the burst of one of the enterprises might delay all the rest.
Security
In durable queues, whatever reaches the bus remains for a time of 60 days and can be retried or replayed:
- Only confirmed events can enter the bus. The receiver checks the signature of each one before sending it to publish [10][11] and it is the only entity that can write to the build-event topic. A consumer should not be the first checker of an event.
- However signatures do not protect against replay. The HMAC does not have a timestamp, which means that a request can be obtained in determination for an unlimited time. We deduplicate according to the delivery ID for the entire retention window.
- Authorize again at the moment of execution. Minutes may pass between acceptance and execution which is why the scheduler checks that the actor has access, that the ref is to the commit, and that the fork policy allows the build.
- Take care of not disclosing secrets in events. Credentials are received from the secrets manager at the moment of execution because events can be replayed or exposed.
Operations
The first sign which appears is consumer lag. Kafka, in this case, directly reflects it [15] and before the errors and latency. The correlation ID, which is passed from the webhook to the status write-back [16], presents the answer to the question "why didn't this build get a status?"
The dead letter queue must come before that one and the problem with the first poison-pill event arises. A message which has a status issue can block everything that is in the partition line for the queue. After a number of retries it is moved to the dead letter queue meaning that there must be its own monitoring and a checking process in the dead letter queue. It means that if the retention period is over sixty days then it is possible for us to repeat the events once we have the fix.
Back pressure. When the execution layer is overloaded or the debounce pool is full then consumers are forced to yellow their partitions and let Kafka take care of the backlog and avoid memory use from consumers.
Limits on service requests. GitHub has set limits on the number of service requests made by a particular organization [17], but both the callback and pull requests can be counted as service requests dependent on those limits. In the case of our particular GitHub application, it can be installed to each organization separately so the requests made by one organization can never affect the limit of another organization.
Data validation. There are some events that are never delivered by GitHub [1], which means the system never receives them. In this way, there is a special job that compares current pushes with the requests already recorded in the system and generates the missing requests.
Trade-offs and Limitations
After a webhook comes in the status check may take a bit to show. Also the event bus becomes a primary dependency. If it goes out the whole pipeline grinds to a halt, not just a single request.
In an organization that runs through a few hundred builds a day this architecture is overkill. At that scale the simple, easy to debug and economical nature of a synchronous design is what is required and a partitioned bus with a consumer group scheduler just adds to the cost and issues without bettering the product. The crossover point is not a set number. It is when you see burst absorption issues, partial failures which aren’t isolated, or order at scale becoming a regular ops issue instead of a what if scenario.
Production Lessons Learned
Rebalance storms are not a theory—they are a real problem. Every time a scaling event happens the consumer group rebalances and consumption stops on the affected partitions until the reassignment finishes [7]. If a scheduler scales frequently it spends most of its time rebalancing instead of actually processing messages. From the outside it looks like the system is stuck even though every consumer is running fine. Tuning the rebalance protocol didn’t help much. What really made a difference was changing how we autoscale—scale up, scale down slowly and make sure there’s a cooldown period between each scaling event.
Migrate customers out in batches, at once we did not do a full scale move of all at once. The old system did not have an event bus; it was the sync design which went before and was the only go to live option for all teams, which also did not allow for down time. Instead of a full scale switch over we put customers into the new pipeline in batches:.
Dependency analysis was done to see what the old service did which documentation didn't report.
We started with a small set of what we thought to be low risk organizations for the batch migration and only grew each batch once the previous one went through without issue.
Also we had a pre determined roll back plan for any batch which may have to be moved back to the old system without having to improvise.
Batching kept the blast radius small. A problem in the new pipeline affected only the organizations already moved and issues from monorepos and high-volume organizations surfaced before they could reach everyone.
Rehearse failures before production. Hot partitions, rebalance storms and poison-pill events are far cheaper to find on purpose than in an incident. Two forms of validation are worth building in. First we will run through a set of anonymized production webhooks which includes bursts on the staging environment because synthetic uniform load does not present the issue at hand which it should. Second we will introduce the failures which the design is intended to weather out[18]:
Kill a consumer in the middle of a batch to see that no event is lost and that redelivered duplicates are rejected by idempotency keys.
Put forth a malformed event to have it land in the dead letter queue instead of causing its partition to stall.
Scale the consumer group up and down to see what amount of lag a rebalance at the current partition count adds.
Conclusion and Outlook
The design of the synchronous webhook is appropriate for the workload that most companies run. However, the trouble starts when groups believe they can scale the webhook by increasing the number of runners. Long before the execution capability becomes a limiting factor, changes need to be made at the ingestion level: before the execution capacity is reached the structure should change: there should be *stateless receivers for verification and publication*, partitioned announcement tool, horizontally scalable groups of consumers for planning the workload, and the collection of undelivered events to control the processing of batched tasks. All of these components are well-known. The trick lies in bringing them together in a way appropriate for the requirements generated by the work of the software (including the requirements for putting the records in their proper order, authorizing access to the information, and isolating failures).
That which we see is coming faster for more teams. We see in industry research a greater distance growing between what organizations do with CI/CD as a core infrastructure as opposed to a utility issue [2] also in play is AI in development which is increasing build volume still. We will put the bus forward as a source of operational signal as we do for a scheduling issue. We will add a structured failure classification field at publish time which will turn each build failure event into a data point for the auto diagnostician, we do away with a separate log scraping pipeline. As more organizations make this transition we expect to see shared and proven tooling for CI ingestion to come out rather than each team having to piece together from general purpose messaging we did.
References
[1] GitHub, "Handling failed webhook deliveries." [Online]. Available: https://docs.github.com/en/webhooks/using-webhooks/handling-failed-webhook-deliveries
[2] DORA, "State of AI-assisted Software Development 2025." [Online]. Available: https://dora.dev/research/2025/dora-report/
[3] GitHub, "Webhook events and payloads." [Online]. Available: https://docs.github.com/en/webhooks/webhook-events-and-payloads
[4] GitHub, "Best practices for using webhooks." [Online]. Available: https://docs.github.com/en/webhooks/using-webhooks/best-practices-for-using-webhooks
[5] Apache Software Foundation, "Apache Kafka." [Online]. Available: https://kafka.apache.org/
[6] Confluent, "Kafka Consumer Design: Consumers, Consumer Groups, and Offsets." [Online]. Available: https://docs.confluent.io/kafka/design/consumer-design.html
[7] Apache Software Foundation, "Consumer Rebalance Protocol," Apache Kafka Documentation. [Online]. Available: https://kafka.apache.org/41/operations/consumer-rebalance-protocol/
[8] Apache Software Foundation, "KIP-429: Kafka Consumer Incremental Rebalance Protocol." [Online]. Available: https://cwiki.apache.org/confluence/display/KAFKA/KIP-429%3A+Kafka+Consumer+Incremental+Rebalance+Protocol
[9] Apache Software Foundation, "KIP-345: Introduce static membership protocol to reduce consumer rebalances." [Online]. Available: https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances
[10] GitHub, "Validating webhook deliveries." [Online]. Available: https://docs.github.com/en/webhooks/using-webhooks/validating-webhook-deliveries
[11] H. Krawczyk, M. Bellare, and R. Canetti, "HMAC: Keyed-Hashing for Message Authentication," RFC 2104, IETF, Feb. 1997. [Online]. Available: https://www.rfc-editor.org/rfc/rfc2104
[12] Confluent, "Message Delivery Guarantees for Apache Kafka." [Online]. Available: https://docs.confluent.io/kafka/design/delivery-semantics.html
[13] Apache Software Foundation, "KIP-98: Exactly Once Delivery and Transactional Messaging." [Online]. Available: https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging
[14] Spring for Apache Kafka, "Manually Committing Offsets," Reference Documentation. [Online]. Available: https://docs.spring.io/spring-kafka/reference/kafka/receiving-messages/ooo-commits.html
[15] Apache Software Foundation, "Monitoring," Apache Kafka Documentation. [Online]. Available: https://kafka.apache.org/documentation/#monitoring
[16] OpenTelemetry, "Context Propagation." [Online]. Available: https://opentelemetry.io/docs/concepts/context-propagation/
[17] GitHub, "Rate limits for the REST API." [Online]. Available: https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api
[18] Principles of Chaos Engineering. [Online]. Available: https://principlesofchaos.org/