Every rig has a first crash in it somewhere: a marginal VRAM chip, a power supply that sags under full load, a riser that works until the chassis warms up. The only question is whether that crash happens during your burn-in test or during a renter's 40-hour training job. One of those costs you an evening. The other costs you your reliability score.
Why burn in before you list, not after
Marketplaces track reliability per machine, and renters see it. On Vast.ai, crashed jobs and unexpected downtime feed a reliability stat that pushes your listing down the search results, and it recovers slowly. A machine that fails in its first week of rentals can spend months digging out of the hole, earning less the whole time.
Burn-in flips the economics. Before listing, a failure costs you nothing: no refunds, no angry renter, no stat damage, just a part to reseat or RMA while the machine is not earning anyway. The same failure after listing is the most expensive kind there is. Spend the day of testing; it is the cheapest insurance in hosting.
Stress the cores with gpu-burn
The standard tool is gpu-burn, a small open source program that hammers every card with continuous matrix multiplications and checks the results for errors. Wrong math under load is exactly the failure mode that ruins renter jobs, so this is the test that matters most.
git clone https://github.com/wilicc/gpu-burn.git
cd gpu-burn
make
./gpu_burn 3600The argument is seconds, so that is a one-hour run across all GPUs. You want to see OK next to every card at the end, with no FAULTY results and no errors counted. It needs CUDA installed to build; if the machine is already set up per our Ubuntu server checklist, you can also run it inside a CUDA container instead of building on the host.
Do not forget the VRAM
Core compute can be flawless while one bad memory chip corrupts a renter's model checkpoints. VRAM errors are sneaky: they often appear only when the memory is nearly full, or only at high memory temperature, which is why short casual tests miss them.
gpu-burnallocates most of the card's memory by default and its result checking will catch many memory faults, especially on long, hot runs.- For a dedicated pass, memtest-style GPU tools (such as the cuda_memtest family) walk patterns across VRAM the way classic memtest86 does for system RAM, and report the failing region.
- Watch
dmesgduring and after testing:sudo dmesg | grep -i xid. Xid errors are the driver telling you about hardware-level faults, and any Xid during burn-in disqualifies the machine until you find the cause.
Watch temperatures for the whole run
Burn-in is also your cooling exam. In a second terminal, log the cards every five seconds:
nvidia-smi -l 5What a passing run looks like:
- Temperature climbs, then plateaus and holds flat for hours. A slow upward creep two hours in means heat is accumulating in the chassis faster than you are removing it.
- Clocks stay steady. Sagging clock speeds at constant temperature mean the card is throttling on power or thermals, and a renter's benchmark will notice.
- Power draw sits near the limit without the machine resetting. A multi-GPU rig at full load is the harshest test your power supply will ever face; better to find its limits now.
If temperatures plateau above roughly 80 C, fix cooling before you list. A custom fan curve holding the cards near 65 C is the usual answer, sometimes paired with a modest power limit.
One hour is a smoke test, not a burn-in
When the test fails
A failed burn-in is a good outcome: the machine told you the truth while it was cheap to hear. Reseat cards and risers, test GPUs one at a time to isolate the culprit, check PSU capacity against total board power, and rerun until a long run passes clean. Keep notes; the same symptoms will mean something if they ever reappear in production. And when a listed machine does misbehave later, work through the 10-minute offline checklist before assuming the worst.