Three research teams. Eight A100s. One Kubernetes cluster. The default scheduler will get you to "always-full queues and a Slack channel full of complaints" in about two weeks. Here is the configuration that has kept things peaceful for ten months.
NVIDIA device plugin: the table stakes
The plugin exposes GPUs as a schedulable resource. Pods request nvidia.com/gpu: 1. This is the minimum.
MIG slicing for inference workloads
A100s support Multi-Instance GPU partitioning. We slice the inference cards into 4x 10GB partitions; small inference jobs get 1/4 of a card and run alongside three peers. Training cards stay whole.
Priority classes that actually preempt
- Production inference: priority 1000, non-preemptible.
- Scheduled training: priority 500, preemptible by inference.
- Interactive notebooks: priority 100, preemptible by anything.
- Idle reclaim: priority 10, runs only on unclaimed slots.
The preemption rules
Lower-priority workloads get killed with a grace period; the workload is responsible for checkpointing on SIGTERM. We provide a small library that wraps PyTorch checkpointing and resumption, and every job that skips it gets killed without ceremony.
Per-team quotas
ResourceQuotas per namespace cap GPU-hours per team per week. The cap is generous; the visibility is the point. Teams can see their consumption in Grafana and start budgeting themselves.
What broke and what we learned
The first month was a series of "interactive notebooks starving everyone else" incidents. The fix wasn't technical. It was telling people their notebook would die after 4 hours, then making it actually die.