Temporal correlation and change attribution in ops ML

The most common question in incident triage is temporal: “What changed near when it broke?”

Humans answer it by lining up deploy markers on a Grafana chart. ML can accelerate that alignment, but only if we are precise about what “correlation” means and what it does not prove.

Symptoms as changepoints

Many SLO and availability symptoms are detected as changepoints in a metric stream: step increases in error rate, latency quantiles crossing a budget, or pool utilization jumping to a new plateau.

Classical approaches include:

• Control charts and rolling z-scores (fast, interpretable).
• Bayesian online changepoint detection (better for gradual drifts).
• Seasonality-aware baselines (critical for business-hours traffic).

The output is not “root cause.” It is a time window where the system behavior became abnormal.

Change events as candidates

Change candidates include deploys, feature flags, certificate renewals, policy rollouts, and infrastructure maintenance. Each has a timestamp and metadata (service, ring, version, actor).

Attribution scoring asks: How plausible is it that a given change explains symptom onset in that window?

A practical score combines:

1. Temporal proximity: change time close to symptom onset inside a learned or configured bound (e.g. deploy within 30 minutes of onset).
2. Topological relevance: change touched a service on the path from alert source to failing dependency.
3. Dose–response plausibility: e.g. connection pool metrics move in the direction a deploy would predict.

This is related to Granger-style thinking (“does knowing the change improve prediction of the symptom?”) without claiming formal causality in a nonlinear distributed system.

Why correlation ≠ causation (and why we still show it)

In a microservice graph, confounders are everywhere: a traffic shift and a deploy may coincide because both follow a marketing launch. ML must rank candidates, not crown winners silently.

Good UX encodes humility:

• “auth-svc v2.14.1 deployed 11m prior” (fact)
• “Linked to DB load spike” (hypothesis with confidence)
• Human can demote the link with one action

Competing hypotheses from time alone

Multiple changes may fall inside the window. Ranking requires marginal contribution:

• Changes on the critical path from alert to failing node score higher than unrelated fleet noise.
• Changes with blast radius matching observed fan-out (one redis cluster, many consumers) score higher than local-only events.

Graph distance turns time alignment from a flat list into a structured search over dependencies.

Failure modes we design against

Pitfall	Mitigation
Alert storm creates fake onset	Deduplicate correlated alerts; anchor on SLO burn
Rolling deploys span hours	Attribute to ring/stage, not global “deploy”
Delayed symptoms (GC, cache fill)	Widen bounds per symptom class from history

Connection to investigate mode

Investigate mode emphasizes hypothesis ready: ranked theories with time-linked evidence, not auto-rollback. Temporal attribution feeds the score; humans choose whether to execute repair playbooks.

That separation is what lets teams trust the science without trusting unchecked automation.