← Journal · · Deep dive
Alternative GPU sharing
Hey everyone! It's been a year and a half since my first article about GPU sharing, where I talked about MIG, MPS, Timeslicing, and how to work with them. By now MIG has become the standard for splitting GPUs for a lot of people, since it's available out of the box and offers great hardware-level isolation!
But we're not standing still, more and more interesting GPU sharing tools keep coming out. Let's take a look in this post at alternative ways to split up GPUs :)
📌 Hacked k8s-vGPU-scheduler, aka Hami-Project On April 7-8 I was at DevOps Conf and heard an interesting talk about using vGPU. Yes, in fact this thing is licensed, you need to pay Nvidia to use it. But by digging into the source of nvidia libraries, the folks from cloud.ru managed to implement this approach themselves without buying licenses. And it turned out that some folks in China were a bit ahead and open-sourced their idea under the name project-HAMi. The project is actually under the CNCF umbrella! Thanks to the guys for sharing this find in their talk!
The first screenshot shows how resource isolation works when you declare it in the manifest. You split the GPU either via vGPU (as in screenshot 2) or via dynamic-mig. They even built a nice dashboard.
📌 Dynamic Resource Allocation DRA fundamentally changes the approach to allocating GPU resources for pods.
Now you declare GPU resources ahead of time, describe them, and attach them in your manifests, which lets you use the same GPU for multiple pods (see screenshot 3). The implementation is available here. Starting with k8s 1.32 there should be dynamic MIG, but I haven't checked it yet.
📌 KAI Scheduler from Nvidia
A native Kubernetes scheduler from Nvidia, open source, for large AI workloads. We can also set the vRAM amount for a specific pod, or even 0.5 GPU (see screenshot 4).
But judging by this issue, under the hood it's just regular timeslicing. And if you have 3 pods each using 0.5 GPU on the same card, you're probably going to hit an OOM 🔥
That's the short overview I ended up with, maybe it'll grow into a full article someday! Have a great day, everyone!