Main Content

Efficient, Workload-Specific Partitioning of Topics in Publish/Subscribe Model

Author: Amr Mahmoud Asad Audi

Supervisors: Prof. Dr. Boris Koldehofe, M. Sc. Ahmad Baghdadi

Format: Master's thesis, TU Ilmenau

The Publish/Subscribe (Pub/Sub) model is a foundational communication paradigm in distributed systems, with Apache Kafka emerging as a leading platform for real-time, high-throughput data streaming. Central to Kafka’s scalability is its topic partitioning mechanism, yet existing strategies either rely on simplistic static heuristics or complex adaptive controllers, neither of which account for workload-specific dynamics. This thesis investigates the problem of efficient, workload-aware topic partitioning with a focus on static, pre-calculated strategies. Building on the optimization framework of Raptis et al., this work extends the model by incorporating explicit workload parameters: message size, message rate, distribution inefficiency (δ), and consumer group heterogeneity (γ). Two complementary heuristics are proposed; BroMin, which minimizes broker usage for efficiency, and BroMax, which maximizes parallelism for throughput. Both are integrated into an automated solver and validated on a six-broker Kafka cluster in KRaft mode. A comprehensive experimental campaign across six scenarios, including baseline validation, heterogeneous consumer groups, burst workloads, distribution inefficiency, and producer- versus consumer-heavy regimes, demonstrates that workload-aware heuristics consistently outperform Kafka’s default partitioning and the original Raptis heuristics. Improvements reached up to 95% higher throughput and 63% lower tail latency, while also achieving greater stability across workloads. The results confirm that static, workload-specific partitioning is both practical and effective, providing deterministic, interpretable, and resource-conscious plans for system architects. This approach bridges the gap between simplistic defaults and complex adaptive methods, offering a foundation for future research on adaptive, multi-tenant, and cloud-native partitioning strategies.

Completed: 15.09.2025

References: https://bibliographie.tu-ilmenau.de/servlets/DozBibEntryServlet?mode=show&id=ilm_mods_00078561