← Journal · · Talk sources
Contributions to Kubeflow Pipelines: links from the talk
As promised, here are the links to our proposals / PRs / issues in Kubeflow Pipelines. The pull requests were done by my colleague Anton Pechenin: Anton, thank you so much!
• proposal for central driver implementation: a proposal for moving KFP from a driver pod to a lighter central driver architecture, to reduce task startup overhead.
• central driver implementation: the practical implementation of the central driver approach via an Argo plugin flow instead of a separate driver pod per task.
• recurring runs queue throughput optimization: optimization of scheduled/recurring runs that reduces unnecessary reconciliations and increases the throughput of the recurring-run queue.
• gRPC metrics to api-server / optimize execution spec reporting: adds metrics to the api-server and improves observability and efficiency of the KFP backend.
• pod lifecycle failure visualization: improves the display and diagnostics of pod lifecycle errors in the UI, so you don't have to dig for such failures only through kubectl.
Separately:
• Kubeflow ML Hub
• Kubeflow Manifests: for anyone who wants to try deploying the platform themselves, you can grab the manifests and charts from there.
I've collected the rest of the sources that can also help when building an ML platform here:
https://t.me/mlops_infra/114