Main Content

Understanding the effect of collocating heterogeneous ML models on GPUs for scaling inference services

Inference serving systems provide users access to the trained DL models by provisioning distributed datacenter resources. Users send requests to the DL services, which infer from user inputs and return predictions. To handle dynamically changing numbers of user requests, inference-serving systems adaptively allocate resources to DL services to keep response times below a threshold. GPUs are the dominant hardware accelerators used by providers for DL inference. Inference serving systems typically share GPUs across multiple tenants (trained DL models) to increase utilization of the costly datacenter GPUs. 

There exist multiple ways to share GPUs. Temporal sharing methods schedule kernels (functions to be launched on a GPU) of different DL models at different times. Spatial sharing methods allow partial allocation of GPU cores (e.g., a percentage) and the simultaneous execution of different DL model kernels on a GPU, leading to better GPU utilization than temporal sharing methods. Spatial sharing techniques, like CUDA MPS and Green Contexts, while isolating processing cores, introduce interference among shared resources, such as GPU memory bandwidth and GPU caches.

The goal of this thesis is to understand how the spatial co-deployment of different ML models (for both CUDA MPS and Green Context) affects their performance on GPUs and to derive criteria for placing and allocating model instances (as a responsibility of inference serving systems) to preserve performance at a cost-efficient level.

For performing the evaluation study, students should provide a prototypical implementation to evaluate the co-inference of DL models and the designed placement strategies. The results need to be documented in writing as part of the thesis document.

Literature:

• M. Salmani, K. Razavi, P. Amthor, and B. Koldehofe, Bridging CPU and GPU Autoscaling for Cost-Efficient Inference Serving, To appear in the Proceedings of the 6th Workshop on Machine Learning and Systems (EuroMLSys ’26), April 27–30, 2026, Edinburgh, Scotland Uk. https:/doi.org/10.1145/3805621.3807643
• Paul Elvinger, Foteini Strati, Natalie Enright Jerger, and Ana Klimovic. 2026. Understanding GPU Resource Interference One Level Deeper. In Proceedings of the 2025 ACM Symposium on Cloud Computing (SoCC '25). Association for Computing Machinery, New York, NY, USA, 687–694. https://doi.org/10.1145/3772052.3772270
• Yingwen Chen, Wenxin Li, Huan Zhou, Xiangrui Yang, and Yanfei Yin. 2024. DeInfer: A GPU resource allocation algorithm with spatial sharing for near-deterministic inferring tasks. In Proceedings of the 53rd International Conference on Parallel Processing (ICPP '24). Association for Computing Machinery, New York, NY, USA, 701–711. https://doi.org/10.1145/3673038.3673091

Contact:

mehran.salmani [at] uni-marburg.de