AI is becoming deeply embedded in the systems engineering teams use to monitor, diagnose and recover production environments. Observability platforms can now correlate logs, metrics and traces in seconds, while automated agents can execute operational tasks that once required hours of manual effort. But as infrastructure becomes more distributed and autonomous, the consequences of a poor decision can spread just as quickly as the benefits of a good one.
Ran Tao has spent his career working at the intersection of cloud infrastructure, high availability and enterprise operations. A Cloud Support Engineer and Fellow of the British Computer Society, he has led more than 1,200 enterprise customer engagements and serves as an accredited subject matter expert in cloud messaging technologies.
In this interview, Tao explains where AI-powered observability delivers real value, why redundancy can still produce cascading failures, and how organizations should test resilience beyond scripted disaster-recovery exercises. He also discusses the risks of unbounded automation, the recurring mistakes companies make when designing high-availability systems, and why human judgment remains essential when engineers are responding to incomplete information and real operational pressure.
- How are AI-powered observability tools changing the way engineering teams detect anomalies and diagnose production failures, and where do these systems still fall short?
Detection can be turned into an exploratory process driven by AI-powered observability. Instead of monitoring against pre-defined static metrics and thresholds for known problems, AI-powered tools can now correlate metrics, logs and distributed traces to surface developing issues. These tools group related alerts, rank them by severity, and in seconds come up with a most likely cause, as opposed to an engineer spending minutes to hours looking for the anomaly by hand. They also help combat alert fatigue, surfacing the issues that affect users rather than every transient spike in the system.
Where AI-powered observability tools fall short is that they might not reliably be able to generate explanations for the alerts they flag. Bounded by the tool’s visibility, they can only reason within the system the engineer’s own team manages. Some anomalies are in fact expected, led by changes from upstream or dependent downstream services that the AI observability tools are never aware of. On the other hand, if a team allows AI to self-improve or self-correct, its model may drift, at which point real symptoms could get accidentally ignored.
- How can engineering teams distinguish between useful AI-assisted operations and automation that simply accelerates a poor operational decision?
A task is only worth automating once it has been fully characterized, with enough detail that an AI agent could follow the work in a deterministic manner. There are a few things I suggest a team specify before handing the operations over to AI: a clear objective with measurement methods and specifications; the preconditions or eligibility detailing when the operation can be executed; and step-by-step instructions a fully competent engineer could follow. AI-assisted operations and automations are meant for expediting or empowering processes that are tedious or could take extremely long hours for human engineers to work through.
If the team’s goal is to reach for an AI adoption number rather than to serve a real operational need, that is the incorrect mindset. AI tools are still not capable of performing work that was never properly scoped, that arises under unprecedented situations, or that carries significant consequences.
For agentic operations, the worst thing that can happen is cascading action, where an agent holding tool permissions propagates one bad decision through several downstream systems before anyone even looks at it. That boundary needs the same controls we put around any other service: circuit breakers, compartmentalization, and a human review gate.
- Distributed systems are often designed with multiple zones, replicas, queues, and failover mechanisms. Why can those layers of redundancy still produce cascading failures rather than preventing them?
Redundancy is usually designed and built at the component level. The part that redundancy does not cover is where system failure arises. A common example is core business logic deployed active-active across regions while client applications still resolve a single hardcoded endpoint. When traffic shifts to the second region, the architecture fails over correctly and the customer still sees an outage, because the reconnection logic was never built or exercised.
Zones and replicas are independent only within a specific scope. A system may still contain single points of failure that are rarely built with redundancy: an identity service, a DNS server, or the cloud service provider itself. Recent global outages show one shared configuration path defeating every redundant region at once.
- Based on your experience leading more than 1,200 enterprise cloud engagements, what recurring mistakes do organizations make when implementing high-availability architectures?
The most common mistake is taking an availability figure that arrives from leadership and engineering around that figure without good justification. Defining availability service by service, starting from the impact tolerance, then the recovery type (backup and restore, pilot light, warm standby, or multi-site active-active) and then cost, is far better. Choosing without that analysis produces either over-engineering or false confidence. Increasing uptime from 99.9% to 99.99% means the error budget for the month goes from roughly 43 minutes to about 4, and might multiply the engineering, operating and maintenance budget significantly.
Additionally, many teams have recovery plans but never test their execution. Real-world examples tell us that a backup that has never been validated, and whose restoration has never been confirmed, is not reliable in a production environment.
- What role does human decision-making play in production incidents, particularly when engineers are responding to incomplete information, conflicting signals, or time pressure?
When there is incomplete or conflicting information, or when an unprecedented incident occurs, AI can rank hypotheses and estimate blast radius, but it cannot own the outcome of its own decision. However good the model is, someone still makes the decision, and that person must accept responsibility and accordingly the liability for it.
The person best suited to that call is the engineer who owns the system. Is the data complete? Is one signal out of sync with another? Is there a known issue that was never documented? Those answers come from design and human understanding of the system, not the telemetry alone.
Human-facing work is also best directed or executed by real people: bringing the right people into the right discussion, deciding what to tell which audience, and building a plan for a crisis nobody anticipated.
- How should organizations test resilience beyond standard disaster-recovery exercises, and what kinds of failure scenarios are most often overlooked?
Disaster recovery exercises are often scripted: everyone already knows what to do. That validates the script, not the system. There are two types of incident-response exercises I often see organizations overlook.
Automated recovery testing: induce failures to see how long recovery really takes and check data integrity against the system’s objectives, including restoring to the last known-good state.
Wargaming: a systematic exercise that tests the key responders during a crisis, namely who they are, what their authority is, and what channels they will use.
Organizations often overlook external or cascading failure scenarios. A regional outage might leave the backup region with double the traffic, and those sudden spikes might further lead to catastrophic failures. As another example, production failures might come in the middle of the night when limited staff is available, and degraded response time might lead to prolonged impact on the overall system.
- Do you expect AI to make distributed systems more resilient over time, or will increasingly autonomous infrastructure make failures harder to understand and control?
AI generally makes distributed systems more resilient, but only where AI is engineered properly. AI solutions should not be trusted to plan out mitigation strategies for problems they have never seen before. When AI tools are tasked with finding creative solutions, especially in a production environment, they can sometimes worsen the original symptom and leave human responders debugging two systems at once.
AI should be managed like other engineering tools. Engineering teams should include change management, staged rollout, and drift detection for any AI components involved. For critical services containing models, have a documented and tested non-AI fallback. And before running runbooks generated by AI, a human should review them. In other words, AI autonomy is not the risk, but unbounded autonomy is.
