Troubleshooting Is an Evidence Problem
When something breaks in production, coming up with the list of potential issues is the easy part. Anyone can name twenty possible causes, and AI can do it in seconds. Nineteen of them are going to be wrong, and nothing in the list highlights which nineteen. Eliminating them takes evidence, and the evidence is in the logs and in the head of whoever was standing on the floor when it happened.
I’ve said twice in this series that troubleshooting still belongs to the developer, and I never said how to do it. Here’s one approach to the process.
“The [XXX] Isn’t Working”
Replace the placeholder and you get the majority of ticket requests for broken systems. On a production line, that sentence can mean almost anything. The line may be locked because too many items in a row came in out of spec, and it’s waiting for a supervisor’s badge. Maybe the operator forgot to start the data capture component when the order started. Everything at the line may be working just fine while nothing reaches the ERP because the network is down. The device that converts sensor signals to serial data may have hung and need its power cycled. Or a sensor’s readings really are jumping around and won’t settle. To the person standing at the line, every one of those is “the [XXX] isn’t working,” although they are five unrelated problems with five unrelated fixes, and most of them have nothing to do with the component the operator initially feels is broken.
One summer at a manufacturing company, the scale readings did go unstable. The weights wandered and would never settle at the precision needed. A program that judges every roll against a two percent tolerance can’t do much with a weight that wanders. Maintenance was brought in to help troubleshoot the problem. They started by verifying the scales were in working order. Then they started narrowing down to the lines that experienced the issues. It was noticed that the night shift rarely experienced it, so they looked at the shift differences and found the issue. The cause turned out to be the big cooling fans that were running due to the abnormal heat and blowing on the scales. The fix was wind barricades, and not one line of code changed.
Hand that symptom to an AI as “serial scale readings unstable,” and you’ll get a respectable list: baud rate, parity, a loose cable, a failing load cell, electrical noise, a timing bug in the read loop. Every item on it is plausible, and the fan isn’t there, because the fan was never in the code or the error log.
Hypotheses Are Cheap
AI is a very good hypothesis generator, and I mean that as praise. The troubles are the various costs of a hypothesis, which are virtually free to produce yet hours to chase. A confidently wrong one is the most expensive kind, because it sounds like a valid answer. It can be very costly when the hypothesis comes from a tool that has never seen your system. Start there and you spend the morning checking parity settings while the fan keeps the scale from properly registering weights. So the order matters, with evidence gathered first and hypotheses second, and in practice that order is a sequence of steps.
The Sequence
Find out what really happened before you start solutioning based on what was reported. In “AI Has Left the Building” I said the initial report will likely be incomplete, and that undersold it. “The scale isn’t working” is the operator’s diagnosis, a conclusion, when what you needed were the symptoms and logs. Ask what they saw on the screen and what they expected to see instead.
Find out when it happens before you ask why. Which line, which shift, which product, and since when will all narrow it down. “Only during summer” is worth more than a stack trace, because it rules out everything that doesn’t care what month it is. Summer set the time frame for when the issue started, since the fans were brought in to combat the extreme heat. The differences between what was observed on separate lines across multiple shifts are what found the culprit. If you can make the problem happen on demand, better still, because then you can watch it.
Read the logs before you form an opinion. Once you have a theory, you read the log looking for it, and you will find it, because logs are generous that way. Change one thing at a time. If you change three things and it starts working, you’ve learned nothing, and you now own two changes you can’t justify and are afraid to remove.
Write down what you saw before you restart anything. With production down, the first job is to get it running again, and that should be fast. Stopping the bleeding and finding the cause are two different jobs, though, and the restart that does the first usually erases the evidence for the second. Then check the evidence itself. A hopper clogged once, and the sensor registered the partial fill as good since it continued filling from the other hoppers before the weight drop. The system recorded that the correct material had gone into the machine when the wrong blend had actually been mixed. This affected the raw material inventories, and the finished product came out with a layer thin or missing, and every record said the run was fine. The data was accurate about what it weighed and wrong about what that meant, which is why now and then the thing to test is the sensor.
Where AI Fits
AI belongs after the evidence. Give it the actual log lines, the conditions the problem happens under, and what you’ve already ruled out, and the list of twenty becomes a list of three. It is better still at the next step, which almost nobody asks it for. Ask it what would tell two candidates apart: “If it’s the cable, what would I see that I wouldn’t see if it’s the load cell?” That question has an answer you can go and check. In “What Is a Developer for Now?” I said AI can only reason about the system you are able to describe to it, and evidence is how you describe it. Then distrust the answer the way you distrust the report, since both are conclusions from somebody who didn’t look.
The fan was never going to be in the prompt, because nobody knew to put it there. AI will hand you every cause it can imagine, and troubleshooting is finding the one that’s real, which only evidence can do.