AI agents are becoming operational teammates. They can ingest alerts, query logs and traces, correlate deployment events, consult runbooks and recommend or perform remediation. Their promise is not merely faster summarization: It is a shorter path from detection to safe recovery. But an agent that can explain an incident is not automatically an agent that should change production. Restarting a stateless pod, rolling back a customer-facing payment deployment and responding to a suspected credential compromise are materially different decisions. Each requires a different amount of evidence, human accountability and execution control. The practical answer is tiered autonomy. Tier 1 automates well-understood and reversible incidents. Tier 2 lets the agent investigate and recommend, while a human explicitly approves the action. Tier 3 keeps people in command for novel, complex or high-impact incidents while the agent accelerates evidence collection and hypothesis testing. The three tiers...
When a production system fails, one of the first questions we usually ask is: What broke? Was it a storage service, a bad deployment, a memory leak, a dependency or a network problem? That is a useful place to start, but it can also narrow the investigation too early. Some production failures happen even when every component behaves as designed; the problem appears only in their interaction. Systems-safety methods give us a useful way to reason about this class of failure, and I ran into a good example in a large-scale provisioning system. When Everything Works and the System Still Fails The system created storage partitions on demand and, as part of that process, applied an access-control policy to every new partition. The policy logic lived in a shared library that already had another user: An operator CLI that engineers used to apply policies manually, one storage instance at a time. For the CLI, the design worked well. Each invocation read the current policy state of one storag...