← Journal · · Note
Why we're building a separate inference platform instead of extending our PaaS
Hey everyone! I published a new article about why we didn't patch our PaaS for ML/LLM, but are building a separate inference platform on KServe.
For context: Avito's PaaS is an internal platform where a developer deploys a microservice in one click, and CI/CD, databases, logs, metrics, and autoscaling are already built-in. It's designed for stateless services in Go that only need CPU and RAM. More details about it in this article.
The short version:
1️⃣ A model isn't "just another microservice". Dynamic batching (GPU utilization can differ by 5-10x), queues with backpressure, hardware affinity, and FP8 - none of this should live in business logic next to HTTP routing.
2️⃣ GPUs don't split well and they're expensive. HPA based on CPU/RAM is useless for LLMs, you need to scale by queue depth and KV cache. And a cold start with a 15-20 GB image and tens of GBs of weights can take dozens of minutes.
3️⃣ Patching PaaS = quarters of development and risk to the platform that runs our entire backend. In reality, the same separate platform would grow inside PaaS, just messier.
4️⃣ Why KServe. We compared it with Seldon (LLM features only in enterprise), Ray Serve and BentoML ("k8s inside k8s"), vLLM Production Stack (locked to one engine). KServe turned out to be the most feature-rich for both ML and LLM, plus it has a fresh LLMInferenceService based on llm-d architecture.
5️⃣ Our own abstraction. We take the ready-made core and add a wrapper as a controller service: this way we don't lock ourselves into a specific open source project, and if we switch the core, users won't notice.
The platform is already running on real traffic for several teams, and the numbers on GPU-hour savings will come in the next posts, so stay tuned)
Go read it🔥 and let me know in the comments: does inference live in your general PaaS or separately?