Main Content

Performance Prediction of Model Inference on Shared GPUs

Author: Amirmohammad Alidoust Moazezi Lahijan

Supervisors: Prof. Dr. Boris Koldehofe, M.Sc. Mehran Salmani

Format: Master's thesis, TU Ilmenau

Deep Learning inference demand is rapidly increasing due to applications in computer vision, natural language processing, and recommendation systems. Although Graphics Processing Units (GPUs) are the primary platform for these workloads, they are often underutilized because single models rarely saturate the hardware. Multi-tenant GPU sharing, such as NVIDIA Multi-Process Service (MPS), improves utilization by enabling concurrent model execution, but introduces resource contention and unpredictable latency. This work builds on the fine-grained modeling of the NeuSight framework and addresses its limitation in capturing latency degradation under CUDA MPS. The proposed extension incorporates the MPS active thread percentage to better account for performance interference across different Deep Learning architectures and GPU generations. The results show that latency increases as GPU share decreases, especially for larger batch sizes, and that the model differentiates between language and vision workloads. However, accuracy declines on unseen hardware due to unmodeled MPS overheads, and errors increase at very low GPU shares, indicating that wave quantization alone is insufficient. Future work includes modeling compute-bound operations, calibrating for consumer GPUs, predicting end-to-end latency for concurrent workloads, and incorporating energy efficiency as an optimization objective.

References: https://bibliographie.tu-ilmenau.de/servlets/DozBibEntryServlet?mode=show&id=ilm_mods_00080849