The Software That Cannot Stop
Most advice about building and deploying software assumes you can shut it down and reboot the computer. You deploy the new version on a quiet night and roll it back if something goes wrong. I wrote software for production lines that run twenty-four hours a day, every day of the year, and very little of that advice lives through a trip to the floor. Reminds me of the military adage about how brilliant plans never survive their first enemy engagement. The production line doesn’t stop for a deployment or a network outage, so the software can’t either. This constraint changes nearly all design decisions, and AI doesn’t know you have it unless you tell it.
The Line Doesn’t Wait
The program at the end of each line weighs every roll, builds the skids, prints the labels, and sends each finished skid to the ERP. If the program stops, the line keeps going, and product keeps coming whether or not there’s software to weigh it. So the design question was always what the operator does in the thirty seconds after something has failed, since avoiding failure altogether was never on offer.
When the network goes down, the program runs on data it already copied to the line PC, queues every finished skid, and sends them all when the connection returns. In “AI Has Left the Building” I asked whether a manufacturing application can continue collecting information locally and synchronize when connectivity returns. The answer is yes, and on a line that never stops, it was the absolute minimum we could ship.
When the scale itself fails, the operator can enter a full skid at the product’s standard weight. The material numbers come out a little off, which beats product stacking up with nowhere to record it. After every order, the local copy of the ERP data is thrown away and replaced whole, with no comparing and no merging. A clever sync is just one more thing that can be subtly wrong on a night when nobody is watching. The simple, straightforward version can be trusted and easily verified. Even startup is built around the line. The program pulls the schedule first and fills in the detail for each order in the background, so the operator can start the next order without waiting for the software to gather the needed data.
Requirements You Only Find Standing There
The program stops weighing when too many rolls in a row come in out of spec. It locks the system and opens a confirmation window for the operator. A supervisor’s badge is required to unlock the system to start weighing. The trigger number of bad weights was initially set at three across all lines. On the lines with robotic load arms, three turned into a constant problem, because the robot boxed rolls into a case and sent it over the conveyor scale so fast that if the operator stepped away, the system was locked before they got back. After a few months of trial and error, we found for those lines the optimal number was twelve. No specification would have produced that. Three is the right answer on paper and twelve is the right answer next to the robot, which is also why the number is a setting on each line and not a constant in the code.
Shift change taught us the same kind of lesson. At every shift change the program resets and makes the incoming crew confirm what’s physically connected, because the last crew’s setup shouldn’t be taken on faith. The reset is a safety net to make sure the current crew can answer for what is actually being produced. It is also a trap, because people rush it and a rushed check is where the mistakes get in. You only learn the second half by watching it happen.
Shipping Without a Window
You rarely get a true maintenance window in 24/7/365 production, so you arrange not to need one. New features ship turned off when the new code is deployed. It does nothing until a feature flag enables it, one line at a time, from a desk, without anyone touching the line PC. If the feature misbehaves, the fix is just reversing the same setting. Releasing code and turning a feature on became two separate events, and only the second one carries risk.
We knew we would have limited time for deploying new versions, so we had the software do a version check between production orders and at shift change. If the version was different than what we had on our deployment server, it did the update automatically. There was generally a ten-minute window between production orders, and since we were already doing a complete reset at shift change, the application restarted at that point and updated itself.
The issue with making an update mandatory is that it will stop older versions from running until they are updated to it. We rarely did it, only when we knew everything was working and could guarantee that the build was complete and ready to go. At times the deployment would get misconfigured or somehow corrupted on the machine, so we would have to go in, do an uninstall, and redeploy the application. While the repair meant we had to have hands on the PC, all of the computers that ran the program allowed for remote connection, so we could monitor it ourselves. The program stabilized after a major rewrite that made every enhancement feature flag driven. I don’t remember ever rolling the software back in the last seven years.
What AI Doesn’t Know to Ask
Ask AI for a sync routine or a deployment script and you’ll get a good one, for a system that’s allowed to stop. It will retry three times and throw an error. It will show the install dialog and sit there waiting for somebody to click OK. When that somebody is up on the mezzanine connecting resin, the deployment hangs and the data is not getting collected. All of that is the sensible default, and the default assumes stopping is an option. But for systems that can’t stop, you need to state that for the AI and ask yourself these questions before any code gets generated.
What does the operator do if this fails right now? If the answer is “call IT,” the design isn’t finished.
What happens to the work done while it’s down? It has to land somewhere and get reconciled later, or it’s gone.
How do I turn this off without deploying anything? If the only off switch is a rollback, you don’t have one.
None of this is an argument for building everything this way. Most software can stop, and should be allowed to, because every one of those answers costs something. The queue, the local copy, and the manual path all have to be built and then understood by whoever supports them. In “Perfect Software” I argued against the retry strategy nobody asked for, and this is the other side of that argument, because on a line, somebody did ask. Know which kind of software you’re building before you ask for the code.
There’s a saying I wrote down for new people: if the system and the floor disagree, the floor wins, but the system will punish you later. The line was running before the software showed up, and it will be running through the next failure, so the software’s job is to keep up.