A power plant failure after 10 years: an agent on a local model, on site with no network
A system I deployed in a power plant ten years ago went down. The voltage dipped somewhere, and the database locked up in the middle of a write. On site I had no network access. The repair was run by an agent on a local model that ran on my Mac. Before the solution went onto the server, the agent rebuilt the whole software on a copy and tested the fix there.
On the Polish podcast “I to się liczy”, Adrian Andrzejczyk said one thing had surprised him. I have not written a single line of code in two years, and it is possible to work that way. I answered with a story from the week before. I had gone to fix a system in a power plant.
The whole conversation is on YouTube, in Polish. This part starts at 33:21.
This is the fifth post from that conversation. Here I go through the fix step by step, with what I said in the studio and what I added after the recording.
What broke after ten years
The system has been running in the power plant for ten years. When I deployed it, all the code was written by hand. Over those years several other companies extended it. There is documentation, but some things inside have changed.
It is a SCADA-type system for managing an electrical switchgear. It shows which switches are on, where the current flows, what its intensity is and what the flow looks like. The US NIST describes SCADA as a system that gathers and processes data and applies operational controls over long distances. It names power transmission and distribution, and pipelines, as typical uses.
I am not naming the site or the system. I am not going into the details of the fault either. In short: the system was connected to a guaranteed power supply, and the voltage still dipped somewhere. There are a lot of writes going on there, and the database locked up in the middle of a write. It had to be restored, secured and prepared for the future.
Fixing a failure in a system like this is one thing. Building new AI features on top of it is another. Three ways to connect AI to a system that is years old, from a middle layer to RPA, I covered separately.
On site: a computer, a cable, no network
It was an emergency call-out. I arrived, plugged my computer into the server with an Ethernet cable and worked without network access, on a local model.
The agent ran on my Mac, in the Pi harness, with the newest Qwen model at the time. Pi is an open harness under the MIT licence. Its authors describe it as “a minimal, extensible agent harness”. Its model library supports any OpenAI-compatible API, including Ollama, vLLM and LM Studio. Qwen's documentation has a separate chapter on running Qwen models locally, with llama.cpp, Ollama and LM Studio among others.
Pi is not tied to one model provider, so it could work with a model from my own computer. Why choosing a harness and choosing a model are two separate decisions is something I covered in a post on agentic CLIs.
This is what I do hands-on: advising on AI strategy and building agents that survive the demo.
First a copy, then the server
The agent quickly found the bug and prepared a fix. It did not test it on the production server in the power plant, though. It set up an environment for itself and rebuilt the whole software there.
After the recording I added how the agent rebuilt the environment. It started with Docker: it set everything up there and checked it. Then it did something similar on AWS, on a real Windows Server machine, so that the copy would be exact. It set up MySQL, loaded the damaged data and recovered new data.
On that copy the agent ran empirical tests, which simply means it checked what had broken and how to fix it. It ran the actual fix. Then it checked that everything was fixed and that nothing else had broken along the way. Only then did a script with the finished fix go onto the server. The whole fix took me about two hours at home, and then three hours on site.
The fix, step by step
Four steps on the copy, one on the server in the power plant
Copy of the environment
- 1Docker: the whole software rebuilt and checked
- 2AWS: a real Windows Server machine, MySQL and the damaged data loaded
- 3Tests: what broke and how to fix it
- 4The fix on the copy, and a check that nothing else broke
Server in the power plant
Agent: the Pi harness, a local Qwen model, a Mac.
The Agentic Architect
Practical patterns, case studies, agentic projects taken apart and AI news. Zero spam, once a week.
You'll get one email to confirm. Unsubscribe with one click. Privacy policy
Confidence instead of poking at the server
That order really gives you validation at a very high level of confidence. What goes onto the production server is not an experiment. It is a script that has already passed on the copy.
That brings it back to where this part of the conversation started. I have not programmed for two years, and I produce a lot of code. What that looks like day to day, with tasks left for agents overnight, I described in a separate post.
I think a person who came in and started poking around on the server in the power plant could have trouble solving it this efficiently and this fast. Especially if they did not know programming, or had not worked with it for a long time.
