Have you ever wondered about all the hardcore processing that goes on behind the scenes in your Uber app to get you to your destination with no interruptions? Today, even a minute of waiting seems like a long lag, but how do these machines provide high speed and quality real-time processing? The answer is simple – streaming platforms.
Initially developed by LinkedIn, Apache Kafka’s real-time streaming data pipelines and applications are the backbone of Uber today. With massive deployments of Apache Kafka, Uber uses the service to process trillions of messages and multiple petabytes of data per day. The team has even called it the ‘cornerstone of the technology stack’.
Apache Kafka empowers more than 300 microservices in Uber and various workflows like pub-sub message buses for passing event data from the rider and driver apps, streaming database changelogs to subscribers and processing data from Hadoop data lake.
Source: Data flows through Kafka pipelines
Let’s have a look at these services in brief:
One of the problems Uber overcame was partition scalability. An individual Kafka cluster at Uber would possibly have more than one hundred brokers. The engineers found a plausible trend of underutilizing Kafka cluster resources that would invariably reduce the efficiency of system resources. The team also realized that the Apache Kafka consumer coding pattern does not handle poison pill messages and non-uniform processing latency for pub-sub message queuing.
Consumer Proxy
Uber has used Kafka to develop a novel push proxy and solve the issues with the Consumer Policy clusters. It fulfils two gaps – presents the consumers with a simple-to-use gRPC protocol to address the mismatch of Kafka partitions during message queuing, and prevents consumer misconfigurations by hiding them from consumer services.
These essentially fetch messages from Kafka using Kafka binary protocol and separately send each message to a consumer service instance. The consumer service further processes these messages separately and forwards the results to the Consumer Proxy cluster. Once the cluster receives the gRPC status code, it aggregates the processing results of the consumer service’s message and commits offsets to Kafka.
Source: Consumer Proxy Architecture
Features of Consumer Proxy
Uber has overcome the pub-sub message queueing system issues by implementing features via a client-side SDK. In addition, the team chose a proxy-based approach.
The engineering team has taken a multiple programming approach with Go, Java, Python, and NodeJS services. While traditionally different services would be written in other languages for the various client libraries, Consumer Proxy makes it possible to implement only one programming language applicable to all services. This approach also makes it easier for the team to manage the 1000 microservices that Uber runs. Since the message pushing protocols remain unchanged, the Kafka team can upgrade the proxy at any time without affecting other services.
Consumer Proxy also assists in limiting the blasting radius of rebalancing storms as a result of the rolling restart. It rebalances the consumer group by decoupling message consuming nodes from the message processing services. The service can eliminate the effects of rebalancing storms itself by implementing its group rebalance logic.
Lastly, the proxy allows the system to consume messages using four nodes and evenly distribute them to all service instances. Traditionally, the Kafka consumer group has only been able to distribute it to 4 cases at most.
Design
Parallel Processing within Partitions
The system achieved parallel processing of messages within a single partition by consuming a batch of messages in the Consumer Proxy cluster and then sending them parallelly to multiple instances of the Consumer Service. The number of Kafka partitions does not limit the number of consumer service instances since the single node can send messages to various consumer service instances.
Out-of-Order Commit
The out-of-order commit from consumer services to Consumer Proxy was introduced to overcome blockage for the consumer when Kafka cannot process messages in a batch. This makes it possible for consumer services to commit a single message to Consumer Proxy, unlike Kafka, where a commit leads to all messages with lower offsets being committed. Consumer Proxy commit will only mark the specific single message as committed.
Consumer Proxy leverages out-of-order commit to fetch several batches of messages, insert them into the out-of-order commit tracker, and send them parallelly to consumer services. Then, once the range of messages is acknowledged, Consumer Proxy commits them to Kafka and repeats the process over again before the previously fetched messages have even been fully committed to Kafka.
Dead Letter Queue
Poison pill messages are those that cannot be processed, or take too long due to non-transient errors. While these generally block consumer cases, in many cases, it would be preferred to mark the messages for special handling and revisit them later. Uber’s dead letter queue (DLQ) topic stores poison pill messages, allows consumer services to send a gRPC error code, and explicitly instructs Consumer Proxy to persist messages to the DLQ topic. These will be marked as “negative acknowledged” in the tracker, and based on their needs, users can ‘merge’ or ‘purge’ these messages.
To wrap the system up, Uber ensures that consumer services receive neither too few nor too many push requests by specified flow control. Flow control mechanisms include Consumer Proxy taking action to adjust message pushing speed after processing the various functions and the presence of a circuit breaker to stop pushing when the consumer service is down.
Uber has developed an accessible use case of the Apache Kafka via its Consumer Proxy.
