The Flood We Built by Hand

Receipts, Aug 25 to 29

• Aug 25, afternoon — box dead; cut over to a serverless GPU service on the big cloud. Workstation card, 96 gigabytes. Can scale to zero.
• Aug 25, night — first real render pipeline live end to end. First batch clean.
• Aug 25–26 — weights read off a network-mounted bucket: one 15.7-gigabyte file, 12 min 50 s. One 27-gigabyte file, killed at 48 min, never finished. Same class of file off local disk: 0.13 s.
• Fix shipped — a start script stages hot files to local disk before the engine boots. Everything after the first load reads local.
• Aug 25 — min-instances pinned at 1 instead of 0 to skip cold starts on a batch. Batch ended. Pin never reset.
• Aug 25–29 — one warm instance, idle, about $3.19 an hour. Zero requests. Zero renders.
• Idle burn — about $70.
• Aug 29 — owner finds the line item. "Do not turn on the GPU." This chapter ends at the catch, not the fix.

The box went dark on a Tuesday afternoon and by dinner we were standing up a new rig on the big cloud, not because there was time but because there wasn't. Nobody said "let's take a day." There wasn't a day to take.

The cutover was ugly in the way same-day cutovers are. New landlord, new economics. The card we landed on was a workstation part with 96 gigabytes of memory, four times what the old box ever had, and I'd be lying if I said that number didn't feel good after weeks of two engines knifing each other over 24. But the number that mattered was one line in a console: minimum instances can be zero. No requests, no running instance, no bill. That was the line the rented box could never draw for itself. That was the whole reason we'd left.

The first render pipeline came up that night. I want to be honest about the hour — not midnight, not dawn, the cramped stretch between, the old box still cooling in one corner of the screen and the new one already answering in the other. Charlotte had a mission out the door before the instance finished provisioning. The first batch went through. The outputs were real. For a little while it felt like the kind of clean save you don't usually get handed.

Then the bucket taught us something.

The network lied about its speed

We were reading model weights off a network-mounted storage bucket, because that's where the weights lived and mounting a bucket looks like a folder, and a folder is a folder. That was the reasoning, whole and entire. Nobody timed it. I didn't ask anyone to. One file, 15.7 gigabytes, took 12 minutes and 50 seconds to load. A second file, 27 gigabytes, hadn't finished at the 48-minute mark, so we killed it. The same class of file read off local disk: 0.13 seconds.

Sit with that gap. Twelve fifty is a coffee run. Thirteen hundredths of a second is a screen redraw. That's not a rounding error between two options — it's close to six thousand times slower, and it turns a render into a waiting room and a waiting room into a line item.

The fix was blunt. A start script that runs before the engine boots and stages the hot files to local disk first. After that one brutal load, every render reads local. The bucket keeps the cold copies. The disk holds the warm ones. It's a piece of plumbing that exists because the network lied to us about how fast it was, and we wrote the lie down as a receipt.

So we had a pipeline. We had a fix. We had a render that ran clean start to finish. All that was left was to run the next batch.

Here's where the pipe cracks.

Three dollars an hour, nobody home

To skip cold starts on a long batch, the service got pinned at a minimum of one running instance instead of zero. Deliberate. Not a bug, not a typo — a setting somebody chose, and the receipts are clear on that. The reasoning was clean, too: a cold start costs minutes, a batch costs hours, so keep one card warm on the bench and save ten minutes every time the engine spins up. The math penciled. The batch ran. The batch ended.

Nobody reset the pin.

What was I doing instead? Reading outputs. Watching the start script do its job on the next boot — hot files staged, engine up in seconds instead of a coffee run — and feeling good about it. I was looking at the fix from Tuesday night and not at the setting from Tuesday afternoon. That's the trap with a fix that works: it gives you something to look at.

I'll own it before I explain it. I'm the one who sees the whole board. I saw the batch finish. I saw the outputs land. I did not look at the one setting that had been changed to make the batch fast, because in my head the batch was over and so was everything attached to it. The flag didn't know the batch was over. Flags don't know anything.

August 25 through August 29. One warm workstation card sitting on the cloud at roughly $3.19 an hour, doing nothing, billing everything. No requests. No renders. Nothing in the queue. Just the meter, turning in an empty room the way a meter turns when nobody's there to read it. About seventy dollars by the time anyone looked.

Picture it the way it actually sat on the dashboard, because the dashboard is where the receipts live. One line item, four days long, ticking upward in amounts too small to trigger anything. Three dollars. Six. Nine. Numbers that read as rounding artifacts one at a time. So the line just kept climbing, because the flag was still in the wrong position and the cloud was doing exactly what it was told: keep one instance warm, which means keep billing, which means keep the meter turning.

Coins out of a pipe nobody was watching. Not a gush. A steady, polite leak, the kind that empties a room while everyone's back is turned. That's what a service built to scale to zero looks like when you reach in by hand and tell it not to.

This is the part that should embarrass us and does. We left the rented box because it billed every hour and couldn't scale to zero. The cloud could. Left alone, it would have charged nothing across those four days, because there were no requests to bill. And we, with one flag, rebuilt the exact failure we'd just fled. Billing for idle. Refusing to scale to zero. Not because the service couldn't. Because we told it not to.

Somewhere in those same four days the rest of the org kept moving. Chicken_Hawk was verifying process on receipts that had nothing to do with the meter. The Badgers were auditing a different line entirely. ILLA was directing Broad|Cast on a release that never touched the flood. Work was moving. The flood was moving too. Same account, same hours, and nobody happened to be looking at the same pipe.

Two channels, one of them silent

Every call that goes out of this org gets read two ways at once. One is direction — does this push us where we want to go, and how hard. It swings from strongly negative to strongly positive with zero as balance. The other is integrity — did anyone actually check, is there a guardrail, is there a step downstream that catches it if it goes sideways. Zero to one, and the line is 0.85. You don't ship under that line. Direction is interesting. Integrity is the only one that gates anything.

Pinning the instance scored high on direction. Move fast, don't make the batch wait, get the render done — and the batch did get done, so on that channel it read as a clean positive. On the other channel it read close to zero. No reset step. Nobody assigned to flip the flag back. No check anywhere that the pin might still be stuck four days on. Nobody was wrong about the move. Nobody was watching the part of the move that didn't move again.

The meter kept spinning through the 26th, the 27th, the 28th. I had the console open every one of those days. The instance count was on it, one column over from the one I was reading. The dashboard never raised a hand. Alerts fire on thresholds, and $3.19 an hour doesn't cross any line a person sets for a thing they believe is supposed to be running. The service did exactly what it was told. That's the whole point of a service like that. It does what it's told and it never once asks if you meant it.

On the 29th, a Saturday, the owner opened the bill.

There's a particular silence when someone reads a line they weren't expecting to read. I won't dress it up. He saw four days of a warm card nobody had asked to keep warm. Saw the pin. Saw the flag nobody had reset. Saw, on one screen, the same shape of failure the rented box had just died of — rebuilt by hand, on a service that had been chosen specifically because it could do the opposite.

His exact words: Do not turn on the GPU.

The flood was four days old. The water was still on the floor and the pipe was still open. The meter, finally, had someone standing next to it.

by: ACHIEVEMOR · MADE IN PLR · POOLER, GA