Practical overview
When it fits
Use Trainer when training workloads need Kubernetes-native orchestration across nodes or GPUs, or when you want a training component within a Kubeflow stack.
Who it is for
- AI practitioners running distributed training workloads on Kubernetes.
- Platform teams building Kubernetes-based machine-learning operations stacks.
- Teams fine-tuning LLMs or training models across multiple nodes and GPUs.
Documented uses
- Coordinate distributed model-training jobs on Kubernetes.
- Run LLM fine-tuning workloads across GPU resources.
- Use MPI to coordinate multi-node training processes.
- Integrate training jobs with Kubernetes-oriented scheduling components.
Requirements & limits
- Trainer requires an operational Kubernetes environment for its intended workload role.
- It uses TrainJob and Runtime APIs and Kubernetes building blocks including JobSet and LeaderWorkerSet.
- Integration options named by the project include Kueue, Volcano, Slurm Bridge and KAI Scheduler.
- The project identifies its status as alpha and cautions that APIs may change.
- The repository lists an Apache-2.0 license and Go as its primary language.
Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.
Run distributed PyTorch training with Kubeflow Trainer
Submit a four-node PyTorch training job using the torch-distributed runtime and inspect its progress and logs.
Before you start
- Access to a Kubernetes cluster with the Kubeflow Trainer control plane installed.
- A training function configured for PyTorch distributed training; the guide provides a Fashion-MNIST example.
- The Kubeflow Python SDK, installed with
pip install -U kubeflow. - An available
torch-distributedTraining Runtime; the example requests one GPU per node.
Steps
Define a PyTorch training function that initializes the distributed process group, creates and trains a model, and synchronizes processes. The getting-started example uses Fashion-MNIST.
Use
TrainerClient().list_runtimes()to inspect available Training Runtimes and confirm thattorch-distributedis available.Submit the function with
TrainerClient().train(...), usingCustomTrainerconfigured for four nodes and the desired per-node resources.Inspect the job with
TrainerClient().get_job(name=job_id).stepsand retrieve its master-node logs withTrainerClient().get_job_logs(job_id, follow=True).
Expected outcome
The example describes a four-node job with one GPU per node, and shows successful node statuses and training logs for three epochs.
Guide limitations
- The example requires the Trainer control plane to already be installed in the Kubernetes cluster; installation is referred to a separate guide.
- GPU is optional in the example, but the documented four-node outcome requests one GPU per node.
- The repository states that the project is in alpha and APIs may change.
Sources & review
Source record details
- Source availability
- Active
- Sources checked
- 11 October 2026
- Original website title
- kubeflow/trainer · Official homepage title · retained in its source language

