One GPU, Twenty Projects: Building a Timeshare Coordinator

7 July 2026 · gpu · scheduling · infrastructure · postmortem

I run one RTX 3090. Against it: a benchmarking pipeline that wants the whole card for hours, a transcription service, an image-segmentation service, video recompression, a voice assistant's LLM, and research training runs that are happy to soak up whatever's left. For months the arbitration mechanism was me, manually, at whatever hour things collided.

So I built the scheduler. It's the same resource-arbitration problem cloud platforms solve, scaled down to a single consumer card — with all the sharp edges intact, because nothing here can assume a well-behaved tenant.

Leases, not hope

Every GPU consumer requests VRAM through a coordinator API before touching the card — a Python helper for code I own, a CLI wrapper for scripts, and a binary shim for tools I don't control (llama.cpp servers, mostly). The coordinator computes effective free VRAM as total minus the max of physically-used and sum-of-reservations. That max matters: a reservation can't be stolen by an unmetered process in the gap between grant and allocation, and a process that allocates more than it reserved can't hide behind its paperwork.

Admission control: a new lease is granted against total − max(used, reserved), never against the optimistic number. Whichever of the two is worse is the one that counts.

Queue position is priority plus waiting time. Priorities run 1 (highest) to 5 (filler), and eviction requires a margin — a holder is only preempted by something meaningfully more important, not by ties.

I learned why aging matters by starving myself. Early on I launched a 22 GB caption-generation backfill at priority 1 — "it's important to me, so, 1" — and it pinned the card while an interactive service timed out behind it, because a priority-1 holder is by definition never evicted. The fix wasn't in the scheduler; the scheduler was doing exactly what I told it. Priority inversion is a policy problem. The batch job was re-classed to 5, and "filler" became a first-class convention: batch work that self-heals — it re-acquires its lease and respawns automatically when preempted — so it needs no babysitting and never holds anything hostage. Filler is the feature that makes a personal GPU feel free: the card is always doing something, and the something always gets out of the way.

Preemption you can take back

The part I'd defend in a design review: preemption is reversible. Instead of killing a victim, the coordinator runs a two-phase pause — ask the process to yield, then verify from NVML that the VRAM actually dropped before marking it paused. Consumers lie; drivers don't. Paused processes checkpoint into RAM under a ledger capped at 32 GiB, so "paused" can't quietly become "the box is swapping itself to death." Pause and resume costs are measured and persisted per consumer, so the scheduler can price a preemption instead of guessing.

The two-phase pause. The middle step is the design's spine: a consumer's "I yielded" is never trusted — the coordinator confirms from NVML that the memory is actually gone before the victim is considered parked.

Reality is reconciled by an enforcer: a 15-second loop that compares the lease table against what's actually on the GPU. An unapproved process gets a tree-aware audit — /proc ancestry plus kernel process accounting, so a shell wrapper can't launder a rogue training run — then SIGTERM with a 300-second grace window, with forensics logged. Anti-thrash immunity stops pause/resume oscillation, and that too was tuned by an incident: a priority-5 filler once held a priority-2 service hostage for ten minutes because restart-tier work had wrongly inherited the 600-second immunity meant for expensive kills. Cheaply-evictable work now gets 120 seconds; only kill-tier keeps 600.

The week every checkpoint was fatal

The war story. Recently, two healthy, checkpoint-parked processes were SIGKILLed back to back, minutes after being parked, in identical sequences. From the outside it looked like "the coordinator admits jobs that can't coexist and the loser dies." The truth was a four-defect chain, and the postmortem is the most instructive artifact the project has produced:

The plan was memoryless about its own executed actions. Once a park completed, the next five-minute replan saw the victim's VRAM freed and the beneficiary running — nothing left to preempt — so it emitted no preemption action. The enforcer read the absence of the action as "thaw now" and woke the victim an hour early, ignoring the slot-end time it had recorded at park time for exactly this purpose.
Thaws didn't check headroom. It tried to resume a 20.5 GB checkpoint into 2.5 GB of free VRAM.
The escalation counter couldn't tell "blocked" from "broken." A strike limit meant to break retry loops on corrupt checkpoints counted every headroom-blocked probe as a failure — five strikes, ~100 seconds, SIGKILL a healthy process.
Free-VRAM accounting double-credited parked memory, so the budget the whole loop reasoned with reported a fully free card while 39.5 GB of live reservations existed.

Six and a half minutes from clean park to SIGKILL. No single step is unreasonable: the planner correctly finds nothing to preempt, the enforcer faithfully reconciles against the plan, the escalation limit does exactly what it was configured to do. The chain is the defect.

Each defect was individually reasonable code. The chain turned a working park-to-fit design into a death timer. And the live re-test after fixing all four caught a fifth, quieter than the rest: the pre-flight probe that checks a checkpoint's state hangs on stopped processes — and every parked victim is stopped by design. The tell was in the event log: strike timestamps spaced exactly 20 seconds apart (a 15-second tick plus a 5-second probe timeout), deterministic across two unrelated process types. That probe had been silently vetoing every legitimate thaw for two days — the event history showed 643 successful thaws, then a hard stop to zero. The data didn't lie; I just hadn't asked it the question.

The signature that unmasked the fifth defect. Endpoints are from the event history (643 successful thaws, none after 4 Jul 19:10); the slope in between is illustrative pacing, not per-day data.

Everything above is now pinned by failure-injection tests — consumers that lie about yielding, processes that die mid-pause, replans that drop actions, leases that outlive their owners. The suite is 742 tests and it earns its keep weekly.

What I'd tell you to steal

Verify state transitions from the substrate, not the participant. Ask NVML whether the VRAM dropped. Ask /proc who the parent really is. A cooperative protocol with adversarial verification is cheaper than either extreme.
Priority inversion is policy, not code. Aging plus eviction margins plus an explicit filler class solved what no scheduler tweak could.
Escalation counters must only count attemptable failures. "Couldn't try" and "tried and failed" are different events; conflating them converts backpressure into a kill signal.
Executed actions need memory. Any planner that re-derives the world from current state will conclude that work already done doesn't need doing — and undo it.
Give your infrastructure a forensics habit. The fifth defect was found by the spacing of timestamps in an event table. Every kill, park, and thaw writes enough to reconstruct why; that's the only reason the postmortem took an evening instead of a week.

It's been in production arbitrating live consumers daily. Single node, single GPU, deliberately — but every homelab with one good card has this exact fight, and an open-source extraction is on the roadmap. If that interests you, say hello.