Anton Alekseev, MLOps engineer: GPUs in Kubernetes, inference, ML platforms. Roadmaps, talks and a journal on ML infrastructure.https://ml-infra.pro/en/enhttps://ml-infra.pro/en/journal/2026-06-29-122/https://ml-infra.pro/en/journal/2026-06-29-122/The text version of my infra.conf 2026 talk about an ML platform on Kubeflow is out. With links to our PRs in the community and manifests, so you can reproduce the platform yourself.Mon, 29 Jun 2026 08:21:20 GMThttps://habr.com/ru/companies/avito/articles/1050838/https://habr.com/ru/companies/avito/articles/1050838/How the ML platform at Avito grew from DevBox setups and per-team unit platforms into a single Kubeflow-based platform, and what problems we hit along the way. I also cover the inference platform and what agentic platforms might look like. Useful both for people building platforms at large companies and for those working with DevBoxes or small unit platforms.Fri, 26 Jun 2026 00:00:00 GMThttps://ml-infra.pro/en/journal/2026-06-23-121/https://ml-infra.pro/en/journal/2026-06-23-121/Speculative decoding from Modal, prefix caching in GKE Inference Gateway, the Databricks AI platform, and Ray on AKS.Tue, 23 Jun 2026 08:09:45 GMThttps://ml-infra.pro/en/journal/2026-06-17-120/https://ml-infra.pro/en/journal/2026-06-17-120/Cloud native is becoming AI native: DRA, Workload API and Inference Gateway, vLLM in Foundry Local, reliable LLM inference at Databricks, and serverless GPUs from Modal.Wed, 17 Jun 2026 07:49:48 GMThttps://ml-infra.pro/en/journal/2026-06-11-119/https://ml-infra.pro/en/journal/2026-06-11-119/We built GPU sharing between ML engineers, similar to Yandex's Dev Cluster, on vanilla Kubernetes + Kueue in Kubeflow. Breaking down which entities it's made of.Thu, 11 Jun 2026 10:12:42 GMThttps://ml-infra.pro/en/journal/2026-06-04-116/https://ml-infra.pro/en/journal/2026-06-04-116/As promised in the talk, links to our proposals, PRs and issues in Kubeflow Pipelines: central driver, recurring runs optimization, metrics, and pod error diagnostics.Thu, 04 Jun 2026 15:41:01 GMThttps://ml-infra.pro/en/journal/2026-05-25-115/https://ml-infra.pro/en/journal/2026-05-25-115/On June 4 at infra.conf I'm talking about how we build an ML platform on Kubeflow at Avito: queues on Kueue, pipeline patches, observability, inference on KServe, and our own model registry.Mon, 25 May 2026 13:12:38 GMThttps://ml-infra.pro/en/journal/2026-04-22-114/https://ml-infra.pro/en/journal/2026-04-22-114/Links by talk topic, and the recording of the talk: approaches to MLOps platforms, Kubeflow profiles, Kueue, Istio Sidecar, distributed training, and LLM inference in KServe.Wed, 22 Apr 2026 13:03:01 GMThttps://ml-infra.pro/en/journal/2026-04-21-113/https://ml-infra.pro/en/journal/2026-04-21-113/A digest ahead of MLechny Put: updates on open source serving projects, async inference via GKE Inference Gateway, and DRANET on AKS.Tue, 21 Apr 2026 13:41:55 GMThttps://ml-infra.pro/en/journal/2026-04-15-112/https://ml-infra.pro/en/journal/2026-04-15-112/The hard part isn't spinning up a model anymore, it's managing it in production: DRANET, Inference Gateway, and an inference operator from AWS.Wed, 15 Apr 2026 08:05:52 GMThttps://ml-infra.pro/en/journal/2026-04-14-111/https://ml-infra.pro/en/journal/2026-04-14-111/The «MLOps for Model Development and Monitoring» course at Yandex Practicum, with a focus on infrastructure: I put together the program core and designed the course infrastructure.Tue, 14 Apr 2026 08:02:31 GMThttps://ml-infra.pro/en/journal/2026-04-09-110/https://ml-infra.pro/en/journal/2026-04-09-110/What's already out there in open source for scheduling AI workloads: scheduler plugins with GPU metrics, and topology-aware scheduling from NVIDIA.Thu, 09 Apr 2026 10:37:06 GMThttps://ml-infra.pro/en/journal/2026-04-08-109/https://ml-infra.pro/en/journal/2026-04-08-109/Launching infrastructure digests: shipping heavy model weights, AI conformance in Kubernetes, multi-node inference on Dynamo, and llm-d in the CNCF Sandbox.Wed, 08 Apr 2026 07:45:13 GMThttps://ml-infra.pro/en/journal/2026-04-06-108/https://ml-infra.pro/en/journal/2026-04-06-108/On April 22 I'm speaking at MLechny Put for the third time, this time about platformization: the path from a DevBox with JupyterLab to a centralized platform, Kueue, Kubeflow, and agentic platforms.Mon, 06 Apr 2026 13:48:19 GMThttps://ml-infra.pro/en/journal/2026-02-16-105/https://ml-infra.pro/en/journal/2026-02-16-105/Why companies go into platformization, how I split ML platforms into Data, ML, and Inference layers. And an open question: do you need it right now?Mon, 16 Feb 2026 14:18:55 GMThttps://ml-infra.pro/en/journal/2025-12-30-104/https://ml-infra.pro/en/journal/2025-12-30-104/Wrapping up the year by track: talks and articles, mentoring and the MLOps course, channel life, and a bit of personal stuff.Tue, 30 Dec 2025 16:58:23 GMThttps://ml-infra.pro/en/journal/2025-12-21-101/https://ml-infra.pro/en/journal/2025-12-21-101/Recorded a podcast with Yura Klassen from Kupper: how to move into MLOps, what a typical workday looks like, what pain points you'll hit, and what skills the market needs.Sun, 21 Dec 2025 14:10:43 GMThttps://ml-infra.pro/en/journal/2025-12-01-98/https://ml-infra.pro/en/journal/2025-12-01-98/Starting a series of posts about inference platforms from the basics: what inference is, how an inference service differs from a web service, and why you should separate business logic from the model.Mon, 01 Dec 2025 15:27:44 GMThttps://ml-infra.pro/en/journal/2025-11-29-97/https://ml-infra.pro/en/journal/2025-11-29-97/Tomorrow I'll be at the career meetup by the self and Zvuk community: we'll talk about careers in ML and MLOps, big tech interviews, and career tracks.Sat, 29 Nov 2025 14:50:50 GMThttps://ml-infra.pro/en/journal/2025-11-29-96/https://ml-infra.pro/en/journal/2025-11-29-96/Posts have gotten rarer, but there's content coming: researching inference platforms for ML and LLM, hitting meetups in Moscow, and writing part two of the article on picking infrastructure for LLMs.Sat, 29 Nov 2025 12:55:19 GMThttps://ml-infra.pro/en/journal/2025-10-04-91/https://ml-infra.pro/en/journal/2025-10-04-91/Thanks for listening to the talk! I open-sourced a lab for picking infrastructure for LLMs, and I'm sharing the pictures that didn't make it into the presentation.Sat, 04 Oct 2025 10:46:36 GMThttps://ml-infra.pro/en/journal/2025-09-30-90/https://ml-infra.pro/en/journal/2025-09-30-90/This Saturday I'm talking at "Ya pro backend" about how I choose infrastructure for LLMs with GenAI Perf. New in the talk: automating the selection with Terraform, the vLLM production stack, and Argo Workflows.Tue, 30 Sep 2025 17:07:04 GMThttps://ml-infra.pro/en/journal/2025-09-17-89/https://ml-infra.pro/en/journal/2025-09-17-89/All sources for the DevOops talk in one message: inference as a web service, faster autoscaling, GPU schedulers, and GPU sharing.Wed, 17 Sep 2025 11:06:24 GMThttps://ml-infra.pro/en/journal/2025-08-29-88/https://ml-infra.pro/en/journal/2025-08-29-88/After a two-month pause, sharing my plans: writing an MLOps course, prepping a DevOops talk on GPU workload allocation in K8s, and releasing part one of the article on picking infrastructure for LLMs.Fri, 29 Aug 2025 09:20:15 GMThttps://habr.com/ru/companies/selectel/articles/941784/https://habr.com/ru/companies/selectel/articles/941784/The first part of a series on choosing infrastructure for LLM inference when a client asks "deploy Qwen for me". We break down what makes up the required VRAM (model parameters, activations, KV cache, buffers), how to pick a GPU and run basic inference. Useful if you are sizing hardware for an LLM for the first time.Fri, 29 Aug 2025 00:00:00 GMThttps://ml-infra.pro/en/journal/2025-06-24-87/https://ml-infra.pro/en/journal/2025-06-24-87/The Evrone channel published the recording of my talk on a private inference platform install. A good reason to remember how the platform is built.Tue, 24 Jun 2025 11:16:26 GMThttps://ml-infra.pro/en/journal/2025-06-06-84/https://ml-infra.pro/en/journal/2025-06-06-84/How tensorizing weights speeds up loading a model into the GPU at vLLM startup. Inside: measurements on Qwen3-8B and Qwen3-32B and config examples.Fri, 06 Jun 2025 10:01:36 GMThttps://ml-infra.pro/en/journal/2025-06-05-83/https://ml-infra.pro/en/journal/2025-06-05-83/I cache HuggingFace weights in S3 for vLLM. Why you can't just dump the HF cache as-is, and the options: rclone, downloading to a local folder, tensorizing.Thu, 05 Jun 2025 07:34:42 GMThttps://ml-infra.pro/en/journal/2025-05-30-79/https://ml-infra.pro/en/journal/2025-05-30-79/We hit 500! To mark it, a video with a DGX B200 cluster and some channel stats: posts, talks, articles, consultations, and podcasts.Fri, 30 May 2025 07:36:12 GMThttps://ml-infra.pro/en/journal/2025-05-29-78/https://ml-infra.pro/en/journal/2025-05-29-78/Sources for my talk at ODS Data Fest, all in one message: inference on Triton, autoscaling and speeding it up, GPU allocation and sharing.Thu, 29 May 2025 15:28:01 GMThttps://ml-infra.pro/en/journal/2025-05-27-76/https://ml-infra.pro/en/journal/2025-05-27-76/On May 29 I'm hosting ODS DataFest at Selectel and talking about inference autoscaling bottlenecks in K8s, GPU sharing, and schedulers for ML workloads.Tue, 27 May 2025 15:14:20 GMThttps://ml-infra.pro/en/journal/2025-05-14-75/https://ml-infra.pro/en/journal/2025-05-14-75/Looking back at Selectel's ML meetup through a co-organizer's article. And if you want a text version of my talk on picking infrastructure, drop some reactions.Wed, 14 May 2025 12:55:29 GMThttps://ml-infra.pro/en/journal/2025-04-23-74/https://ml-infra.pro/en/journal/2025-04-23-74/Sources for the talk on picking infrastructure for LLMs: VRAM estimation, inference frameworks, vLLM configuration, and load testing tools.Wed, 23 Apr 2025 15:32:41 GMThttps://ml-infra.pro/en/journal/2025-04-18-72/https://ml-infra.pro/en/journal/2025-04-18-72/I got hands-on with KAI Scheduler, the open source scheduler from Run:ai. Breaking down the entities: queues with quotas, elastic workloads, priorities, and GPU sharing.Fri, 18 Apr 2025 14:20:43 GMThttps://ml-infra.pro/en/journal/2025-04-17-71/https://ml-infra.pro/en/journal/2025-04-17-71/A guide article on a repo of Triton Inference Server tutorials is out: a demo platform, deploying different model formats and LLMs, auto-configuration, and simple UIs.Thu, 17 Apr 2025 13:32:16 GMThttps://habr.com/ru/companies/selectel/articles/901358/https://habr.com/ru/companies/selectel/articles/901358/A guide to a repository of recipes for NVIDIA Triton Inference Server: an inference platform demo with autoscaling and canary deployments, running models in different formats and popular LLMs, and tuning the configuration with Model Navigator and Model Analyzer. Handy if you are building your own Triton-based inference in Kubernetes.Thu, 17 Apr 2025 00:00:00 GMThttps://ml-infra.pro/en/journal/2025-04-16-70/https://ml-infra.pro/en/journal/2025-04-16-70/That's the title of my talk at the Selectel ML meetup on April 23. And from the picture you can guess which franchise the talk will reference, and what I think about TensorRT-LLM.Wed, 16 Apr 2025 09:01:19 GMThttps://ml-infra.pro/en/journal/2025-04-10-66/https://ml-infra.pro/en/journal/2025-04-10-66/MIG, MPS, and TimeSlicing aren't the only options. Looking at alternatives: HAMi with vGPU and dynamic-mig, DRA, and sharing in KAI Scheduler.Thu, 10 Apr 2025 07:29:02 GMThttps://ml-infra.pro/en/journal/2025-04-02-65/https://ml-infra.pro/en/journal/2025-04-02-65/On April 23 I'm hosting the Selectel ML meetup in St. Petersburg and giving a talk on picking infrastructure for LLMs: GPUs for inference, vLLM and SGLang configuration, and load testing.Wed, 02 Apr 2025 11:43:48 GMThttps://ml-infra.pro/en/journal/2025-03-24-64/https://ml-infra.pro/en/journal/2025-03-24-64/On March 27 in Moscow I'm talking about inference autoscaling in K8s: scale to zero and scale from zero, image caching and pull acceleration.Mon, 24 Mar 2025 09:57:00 GMThttps://ml-infra.pro/en/journal/2025-03-20-63/https://ml-infra.pro/en/journal/2025-03-20-63/NVIDIA released open source Dynamo for multi-GPU LLM inference with vLLM, SGLang, TensorRT-LLM, and mistral.rs backends. Haven't tested it myself yet, but benchmarks are coming.Thu, 20 Mar 2025 18:06:14 GMThttps://ml-infra.pro/en/journal/2025-03-17-62/https://ml-infra.pro/en/journal/2025-03-17-62/A guideline from a consultation: how to deploy inference on a GPU VM, from NVIDIA architectures, drivers, and CUDA to Docker images and picking an inference server.Mon, 17 Mar 2025 08:52:15 GMThttps://ml-infra.pro/en/journal/2025-02-26-61/https://ml-infra.pro/en/journal/2025-02-26-61/A new episode of the "Segodnya na retro" podcast is out with me as a guest: we talked about personal brand, how speaking helps at work, and whether companies should invest in developer speakers.Wed, 26 Feb 2025 16:42:04 GMThttps://ml-infra.pro/en/journal/2025-02-19-60/https://ml-infra.pro/en/journal/2025-02-19-60/The market itself doesn't quite know what MLOps is. I explain it with a diagram: three entities (Data, ML, and Inference), specialist verticals, and a horizontal value-delivery pipeline.Wed, 19 Feb 2025 10:34:29 GMThttps://ml-infra.pro/en/journal/2025-02-12-58/https://ml-infra.pro/en/journal/2025-02-12-58/Found my niche (ML infrastructure), so I'm renaming the channel again. Sharing my research plans for the year and what kinds of questions you can bring to me.Wed, 12 Feb 2025 08:03:02 GMThttps://ml-infra.pro/en/journal/2024-12-31-47/https://ml-infra.pro/en/journal/2024-12-31-47/Looking back at 2024's talks, from DevOps Conf and the ML meetup to PyCon, TechDay, and HighLoad++, and sharing plans for next year.Tue, 31 Dec 2024 14:22:10 GMThttps://ml-infra.pro/en/journal/2024-12-28-45/https://ml-infra.pro/en/journal/2024-12-28-45/The HighLoad++ recordings are out: my talk about the Inference platform is now available, along with the slides and Q&A.Sat, 28 Dec 2024 11:02:11 GMThttps://ml-infra.pro/en/journal/2024-12-27-44/https://ml-infra.pro/en/journal/2024-12-27-44/Wrapping up the year with a second part on Habr: how we implemented autoscaling, canary deploy, the inference graph, and automatic Triton configuration tuning.Fri, 27 Dec 2024 08:09:26 GMThttps://habr.com/ru/companies/selectel/articles/867972/https://habr.com/ru/companies/selectel/articles/867972/The second part about the Selectel inference platform: five features on top of standard Triton in Kubernetes. Canary deployments, autoscaling with faster image pulling, an inference graph, Triton optimization and a UI that needs no developers. For those who need more than just deploying a Triton Helm chart.Fri, 27 Dec 2024 00:00:00 GMThttps://ml-infra.pro/en/journal/2024-12-18-43/https://ml-infra.pro/en/journal/2024-12-18-43/Found a calculator that computes how much GPU memory an LLM needs for training or inference, both for off-the-shelf models and for your own parameters.Wed, 18 Dec 2024 12:51:04 GMThttps://ml-infra.pro/en/journal/2024-12-16-42/https://ml-infra.pro/en/journal/2024-12-16-42/The first part, based on the HighLoad talk: platform requirements, why we started with Seldon and moved away from it, and what the platform looks like now.Mon, 16 Dec 2024 08:42:47 GMThttps://habr.com/ru/companies/selectel/articles/866256/https://habr.com/ru/companies/selectel/articles/866256/The first part of the story of the Selectel inference platform: what requirements we had, why we started with Seldon Core and ended up with NVIDIA Triton, and how the platform infrastructure and its delivery to clients work. Useful if you are choosing what to build your inference on.Mon, 16 Dec 2024 00:00:00 GMThttps://ml-infra.pro/en/journal/2024-12-02-41/https://ml-infra.pro/en/journal/2024-12-02-41/Sources for my HighLoad talk in one message: Istio and canary deploy, autoscaling, faster image pulling, GPU sharing, and more.Mon, 02 Dec 2024 07:40:27 GMThttps://ml-infra.pro/en/journal/2024-11-27-39/https://ml-infra.pro/en/journal/2024-11-27-39/Talking at HighLoad++ about building an Inference platform on Triton: why we moved away from Seldon, canary deploy, GPU node autoscaling, model chains on Ray, and a UI without frontend engineers.Wed, 27 Nov 2024 14:41:09 GMThttps://ml-infra.pro/en/journal/2024-11-15-36/https://ml-infra.pro/en/journal/2024-11-15-36/At KubeCon they gave a detailed talk about DRA for GPUs: native sharing via TimeSlicing and MPS is already available, and dynamic MIG is promised for 1.33.Fri, 15 Nov 2024 08:46:46 GMThttps://ml-infra.pro/en/journal/2024-11-06-35/https://ml-infra.pro/en/journal/2024-11-06-35/Needed an image-selection form when creating an instance in JupyterHub. Instead of jinja2 templates, found a simpler solution: profileList with unlisted_choice in kubespawner.Wed, 06 Nov 2024 10:00:37 GMThttps://ml-infra.pro/en/journal/2024-10-30-33/https://ml-infra.pro/en/journal/2024-10-30-33/Figuring out whether you can register GPUs in Kubernetes under the model name, like MIG partitions. Spoiler: only by patching the device plugin, and nodeSelector helps you pick the card.Wed, 30 Oct 2024 13:53:53 GMThttps://ml-infra.pro/en/journal/2024-10-11-32/https://ml-infra.pro/en/journal/2024-10-11-32/At Tech Day we announced the Inference platform based on NVIDIA Triton, which I'm developing. Sharing the recording of the product talk and an announcement of the technical one at HighLoad++.Fri, 11 Oct 2024 14:30:43 GMThttps://ml-infra.pro/en/journal/2024-10-08-31/https://ml-infra.pro/en/journal/2024-10-08-31/This Thursday I'm talking at TechDay about our new product, an inference platform built on NVIDIA Triton: model lifecycle, autoscaling, canary deploys, and an inference graph on Ray.Tue, 08 Oct 2024 14:45:07 GMThttps://ml-infra.pro/en/journal/2024-09-28-30/https://ml-infra.pro/en/journal/2024-09-28-30/Attended AI Conf by Ontico and wrote down the talks I remember: testing LLM applications, crowdsourcing data labeling, GPT in model training.Sat, 28 Sep 2024 11:25:33 GMThttps://ml-infra.pro/en/journal/2024-09-18-29/https://ml-infra.pro/en/journal/2024-09-18-29/Based on the webinar: I walk step by step through how resource autoscaling with GPUs works in the cloud, and show a practical setup with vLLM and GPT-2.Wed, 18 Sep 2024 16:58:11 GMThttps://habr.com/ru/companies/selectel/articles/844026/https://habr.com/ru/companies/selectel/articles/844026/How GPU inference autoscaling works in Kubernetes: HPA, the node autoscaler, GPU Operator, and why image pulling slows scaling down. In practice we deploy GPT-2 on vLLM, build custom metrics with Prometheus Adapter and load-test it. Useful if you are preparing ML services for peak traffic.Wed, 18 Sep 2024 00:00:00 GMThttps://ml-infra.pro/en/journal/2024-09-10-28/https://ml-infra.pro/en/journal/2024-09-10-28/Inviting you to a Kubernetes meetup where this time I'm in the audience: talks from Selectel, Flant, Magnit Tech, and Hilbert Team. Let's think about how to reuse this in ML infrastructure.Tue, 10 Sep 2024 09:50:06 GMThttps://habr.com/ru/companies/selectel/articles/839528/https://habr.com/ru/companies/selectel/articles/839528/Why setting up a GPU on a server turns into a puzzle of framework, driver and OS kernel compatibility, and how NVIDIA GPU Operator automates it in Kubernetes. We look at running multiple driver versions, precompiled driver containers and the Mandela effect in nvidia-smi, plus how we use the operator for autoscaling and GPU sharing.Fri, 30 Aug 2024 00:00:00 GMThttps://ml-infra.pro/en/journal/2024-08-29-26/https://ml-infra.pro/en/journal/2024-08-29-26/Webinar in an hour: setting up a k8s cluster to work with GPUs, deploying GPT-2 inference, node autoscaling by traffic, and splitting GPUs with MIG.Thu, 29 Aug 2024 11:56:56 GMThttps://ml-infra.pro/en/journal/2024-08-27-25/https://ml-infra.pro/en/journal/2024-08-27-25/In Kubernetes 1.31 you'll be able to mount another container into a pod as a volume. For ML, that's a way to version model weights and configs in a container registry.Tue, 27 Aug 2024 09:20:37 GMThttps://ml-infra.pro/en/journal/2024-08-17-24/https://ml-infra.pro/en/journal/2024-08-17-24/Express42 published the State of DevOps 2024 results. Wrote down what caught my eye: the cloud market, AI assistants, managed services, and DevOps engineer skills.Sat, 17 Aug 2024 13:03:50 GMThttps://ml-infra.pro/en/journal/2024-07-24-23/https://ml-infra.pro/en/journal/2024-07-24-23/Ahead of a PyCon talk about the GPU Operator, I collected links on where to check the dependencies between GPU architecture, driver, CUDA, OS kernel, and framework versions.Wed, 24 Jul 2024 10:04:39 GMThttps://ml-infra.pro/en/journal/2024-06-27-21/https://ml-infra.pro/en/journal/2024-06-27-21/A written version of the talk from the ML meetup: an ML system by analogy with manufacturing, picking a configuration for ML workloads, finding bottlenecks, and using GPUs efficiently.Thu, 27 Jun 2024 13:02:35 GMThttps://habr.com/ru/companies/selectel/articles/822651/https://habr.com/ru/companies/selectel/articles/822651/A text version of my talk at the Selectel ML meetup: what ML systems can borrow from factory assembly lines, how to pick the minimal configuration for an ML workload, and how to find bottlenecks with Goldratt's theory of constraints and a profiler. And then how to use GPUs efficiently in training and inference.Thu, 27 Jun 2024 00:00:00 GMThttps://ml-infra.pro/en/journal/2024-06-21-20/https://ml-infra.pro/en/journal/2024-06-21-20/Back from a Rammstein concert. While I prep new content, you can listen to a podcast where I talk about my DevOps experience and the tasks I run into.Fri, 21 Jun 2024 14:04:27 GMThttps://ml-infra.pro/en/journal/2024-06-09-19/https://ml-infra.pro/en/journal/2024-06-09-19/Heading off on vacation and leaving a collection for DevOps: awesome-lists, roadmaps, guides. Sources were gathered with colleagues, and ChatGPT helped structure it.Sun, 09 Jun 2024 04:38:53 GMThttps://ml-infra.pro/en/journal/2024-05-26-17/https://ml-infra.pro/en/journal/2024-05-26-17/Continuing to dig into pulling and deploying images in containerd: trying nerdctl, eStargz, Nydus, SOCI, and zstd on my own images. Spoiler: my favorite is zstd.Sun, 26 May 2024 10:37:17 GMThttps://ml-infra.pro/en/journal/2024-05-15-16/https://ml-infra.pro/en/journal/2024-05-15-16/My article on GPU sharing in Kubernetes made the Technotext 2023 shortlist. A quick recap: several pods on one GPU and autoscaling inference on MIG.Wed, 15 May 2024 15:08:34 GMThttps://ml-infra.pro/en/journal/2024-05-08-15/https://ml-infra.pro/en/journal/2024-05-08-15/Digging deeper: how image pulling and extraction actually work, and how lazy layer loading via Stargz Snapshotter speeds up service startup.Wed, 08 May 2024 10:55:41 GMThttps://ml-infra.pro/en/journal/2024-05-08-14/https://ml-infra.pro/en/journal/2024-05-08-14/ML images weigh 10-15 GB, so a new replica during autoscaling takes ages to come up. I try a caching registry and find out the bottleneck is layer extraction.Wed, 08 May 2024 10:55:12 GMThttps://ml-infra.pro/en/journal/2024-04-24-13/https://ml-infra.pro/en/journal/2024-04-24-13/Ahead of Selectel's Admin meetup, which I'm hosting, I share my approach to talks: story first, slides second. Plus the books and exercises that helped.Wed, 24 Apr 2024 15:09:42 GMThttps://ml-infra.pro/en/journal/2024-04-18-9/https://ml-infra.pro/en/journal/2024-04-18-9/I'm speaking at MLechny Put (Selectel's ML meetup) today. Sharing the stream and the sources I used to prepare: ML system diagrams, MLOps principles, books on lean manufacturing and theory of constraints.Thu, 18 Apr 2024 15:05:08 GMThttps://ml-infra.pro/en/journal/2024-04-17-8/https://ml-infra.pro/en/journal/2024-04-17-8/Starting a series 'from manufacturing automation to DevOps': I go through lean manufacturing principles using a software product and a coffee plant as examples.Wed, 17 Apr 2024 19:21:12 GMThttps://ml-infra.pro/en/journal/2024-04-04-6/https://ml-infra.pro/en/journal/2024-04-04-6/A quick rundown of three ways to split an NVIDIA GPU between workloads: how TimeSlicing, MPS and MIG differ in GPU support and memory isolation.Thu, 04 Apr 2024 11:36:24 GMThttps://ml-infra.pro/en/journal/2024-04-04-4/https://ml-infra.pro/en/journal/2024-04-04-4/Starting my own channel: I'll share research on DevOps and MLOps, thoughts on tools, and extra material for Habr articles and talks. All in plain language.Thu, 04 Apr 2024 08:39:38 GMThttps://habr.com/ru/companies/selectel/articles/803883/https://habr.com/ru/companies/selectel/articles/803883/How our DataML products team at Selectel moved away from ClickOps and automated the deployment of a cloud ML platform with GitLab and Terraform: from downstream pipelines to a monolithic Terraform job, then splitting it up with remote state, plus automated tests. Useful if you want to stop being afraid of deploying to prod.Thu, 04 Apr 2024 00:00:00 GMThttps://habr.com/ru/companies/selectel/articles/776132/https://habr.com/ru/companies/selectel/articles/776132/Can you repartition MIG on the fly while the GPU is already under load? We look at solutions from run.ai and Nebuly, test dynamic MIG in Kubernetes with experiments, and compare time-slicing, MPS and MIG on the same benchmark. For those who already share GPUs and want to do it more flexibly.Fri, 24 Nov 2023 00:00:00 GMThttps://habr.com/ru/companies/selectel/articles/756934/https://habr.com/ru/companies/selectel/articles/756934/The follow-up on GPU sharing, this time in Kubernetes: how GPU Operator splits GPUs with MIG and time-slicing, and how to set up load balancing, monitoring and autoscaling with Prometheus Adapter and HPA. By the end of the evening we have a prototype of an autoscaling inference platform.Wed, 30 Aug 2023 00:00:00 GMThttps://habr.com/ru/companies/selectel/articles/748544/https://habr.com/ru/companies/selectel/articles/748544/A hands-on guide to GPU sharing: why "one GPU per container" is inefficient and how CUDA streams, time-slicing, MIG, MPS and vGPU differ. Then we set up MIG in practice, run inference servers on the partitions and measure throughput. For those who want to share GPUs between workloads and colleagues.Tue, 18 Jul 2023 00:00:00 GMT