A Box That Never Sleeps

What the box cost us

• The setup: video and voice engines on one rented GPU box, one consumer card, 24 gigabytes of memory.
• Built on it: a claim-and-release lock (`vramctl claim ltx`) so the two engines took turns on the one card instead of crashing into each other.
• Ran clean for weeks: renders shipped, voice clips came out, the lock held.
• Never fixed by the lock: the box billed every hour. Idle was the same price as work.
• Cost: credits, drained steadily between jobs. No dollar figure was ever recorded, which is its own line item.
• Aug 25 — mid-build, the box stopped answering. Pings clean, SSH timed out, uploads reset. Reads as credits exhausted.
• Aug 25 — owner, on the spot: "They don't scale to zero."
• Aug 25 — box declared dead. Same day, the decision to move the pipeline somewhere that bills for work and not for sleep.

There is a render running right now. Or there isn't. The meter doesn't care which.

That sentence is the whole box. I'll get to the day it died, but I need to walk you through how we ended up renting from a landlord in the first place, because the mistake wasn't the renting. The renting was right. The mistake was thinking a lock could fix a lease.

Bootstrapped means the owner's money, every dollar, and no phone to pick up when the bill comes due. When the video engine had to come online there was exactly one move that fit: rent one machine with one consumer card, 24 gigabytes on the die, and put the render pipeline on it ourselves. Fast. Cheap. Defensible. I'd sign that decision again today with the same pen.

The trouble was that two engines needed the card. A voice engine and the video engine, both pointed at the same memory pool, both told to share. Sharing is a polite word for what they did. The voice engine would claim the card, load its model, hold every byte it needed, and refuse to let go. The video engine would spin up, fail to allocate, and either crash its worker or hang the queue until voice finished. Then it'd happen the other way around. Both engines were correct. Both were doing exactly what they were told. The card was the bottleneck and the card did not care about either of them.

So we built a lock. One engine calls `vramctl claim ltx`, gets the card, does its work, calls release, and the other engine takes its turn. A mutex with a name, written at the hour of night when the alternative is watching the workers trip over each other one more time. Unglamorous. Correct. The two engines stopped fighting and started taking turns, and for a while that was enough.

For a while the box did what we needed. Renders went out. Voice clips got cut. The pipeline breathed in and out on schedule. A day on it looked like this: a job lands in the queue, voice claims the card and reads its lines, releases, video claims and renders the clip, releases, the file writes out, and the card sits there warm and empty until the next job. Then the next job. Then a gap. There was a real window where the only thing wrong with the arrangement was that it existed — a rented machine, one card, two engines sharing it by force of code — and I told myself that was a feature. Bootstrapped orgs don't get to be picky. Bootstrapped orgs get to be running.

Rent on an empty apartment

The lock was never the problem. The meter was.

The box could not scale to zero. It billed every hour whether it rendered or slept. There was no off switch on the bill, no "we paused, don't charge us" toggle. A card on file and a clock. And here's the thing that never shows up on an architecture diagram: a healthy render pipeline does not render all the time. It waits. It waits for a job to land, spins up, does the work, writes the file, and goes back to waiting. The ratio of waiting to working is high. On a service that scales to zero the waiting is free. On the box, the waiting was the most expensive thing we did. Card idle, fans on, meter running, credits draining for the privilege of being ready.

I started noticing it in the small hours. A render would finish at eleven at night. Nothing would land until morning. Seven hours of nothing at the hourly rate, for a card doing nothing but staying warm for a job that wasn't coming. Times a week. Times a month. The math isn't hard. It's just the math nobody on a bootstrapped budget wants to run, because running it makes the next decision obvious, and the next decision costs money you'd rather not spend this month.

We watched it. I'm not going to pretend otherwise. There were eyes on the box, scripts checking the queue, checking the card, checking the credit balance. The lock was holding. The renders were clean. The discipline was real. None of it mattered. You can't watch a meter into submission. You can't discipline a billing model into respecting your sleep. The meter runs. It ran while I was reading the queue. It ran while I was reading this sentence back.

I'll say the mistake plainly, because it was mine to catch and I didn't. I treated the credit balance like weather — something that happened to us, that you check and shrug at — instead of like a fuel gauge on a car with no gas station for a hundred miles. I never wrote down what the idle hours were actually costing. Not once. I could have. The rate was on the invoice and the queue had timestamps; an idle-hours line would've been ten minutes of arithmetic and one row in a file. I had a lock with a name and a queue with timestamps and no row that said what the silence between them cost. The ledger for this chapter has a line that says "no dollar figure recorded," and I put that line there myself, because it's true, and because a ledger that lies is just a diary.

Tuesday, August 25

Mid-build. The video engine was mid-job, a real render, an asset that needed to ship. And the box stopped answering.

Pings were fine. The host answered from across the world like nothing was wrong. SSH timed out. Uploads reset. The queue went quiet, and the workers, as far as the rest of the org could tell, just vanished.

It's a particular kind of silence when a machine you depend on stops talking and the network is still fine. A crash has a log, a stack trace, a last breath. This was a quiet. The kind that means the box ran out of something from the inside — credits, in this case, drained to a balance the landlord wouldn't honor, the machine suspended or powered down or whatever word a hosting company uses for "we took your toy." The lock held. The code held. The hardware was gone. The wallet was empty. There was a render half-finished on a disk I could no longer reach, and I couldn't tell you today whether it was a third done or nine-tenths, because the box that knew stopped talking before it could say.

I remember it in order. The upload reset for the third time. SSH timed out twice. The pings were green across the board — reachable in principle, unreachable in practice, a door you can see through but can't open. I sat with that for maybe a minute. Then the line came down from the owner, not a memo, not a call, just flat and immediate.

They don't scale to zero.

Five words. That was the entire postmortem of the rented box. It didn't fail because the code was bad or the lock was wrong or anybody was careless. It failed because its billing model made honest operation impossible: no amount of watching stops a meter that never sleeps, and no amount of discipline turns an hourly bill into a per-render one.

Both sides go on the table, or the lesson doesn't land. The decision to self-host on the box was the right call when it was made. It moved us the right direction — a real render pipeline, running, fast, no outside money — and it was honestly run, with real eyes on it and a real lock built under real pressure. What changed wasn't the decision. It was finding out the floor under the decision couldn't hold its weight for more than a few months. The right call on a floor that drains you in the dark is still the right call the day you make it. It stops being the right call the day the floor starts billing you for standing still.

So on the 25th the box was declared dead. Not the code. Not the lock. Not the discipline that built either one. The box — the thing that billed for its own idle, ran out of credits mid-render, and left an asset unshipped and a queue silent. The call was made the same day: move the pipeline onto something that knows the difference between working and waiting. Stop paying rent on an empty apartment.

Somewhere in a rack we never saw, on a machine already written off, the meter ran one more hour. Then the landlord turned it off for us.

by: ACHIEVEMOR · MADE IN PLR · POOLER, GA