Zabbix Says It's Up

The calls started before I had the laptop open. Order entry was hanging on save. Somebody in the warehouse had already given up and gone back to writing things down on paper, which in that building is the escalation path that outranks the ticketing system.

Every check I owned was green.

What green actually meant

Zabbix was not wrong. It was answering the question it had been asked, and the question was whether the host was alive and whether the service was running. The agent responded. The port accepted connections. The service reported itself as running. Three true statements, none of which had anything to say about whether a person could save an order.

That gap is obvious in hindsight and invisible at the time. Host-up and service-healthy are different claims, and both of them are different again from work-is-getting-done. A monitoring system that cannot express "up but not working" will keep telling you the truth all the way through an outage.

What the database knew

The answer was one layer down, in the place I should have looked first. SQL Server keeps a running account of what it is doing and will hand it over if you ask:

SELECT session_id, blocking_session_id, wait_type, wait_time, command
FROM sys.dm_exec_requests
WHERE blocking_session_id <> 0;

There was one long-running write holding locks, and behind it a queue of sessions doing precisely what they had been told to do, which was wait. Navision was not failing. It was being patient, and patience from a user's chair is indistinguishable from broken. Nothing errored. Nothing crashed. Nothing tripped a threshold, because no threshold I had configured was watching anything that had changed.

That is the part worth sitting with. The outage produced no failures. It produced a queue.

Joining the two views

The fix was not a new tool, it was refusing to keep the two views apart. Host health lived in one console and query-level behaviour lived in another, and the only thing joining them was whoever happened to be looking at both. So they went onto one dashboard: service state and host health where they had always been, and next to them the things the DMVs could answer — how many sessions are blocked right now, how long has the oldest blocker been running, what is the longest current wait.

None of that is exotic. It is the same data any DBA would pull by hand during an incident. The difference is that it is on the screen before the incident, next to the green checks, where it can contradict them.

The next time it happened, the host checks were still green and the blocked-session count was climbing beside them, and the argument about whether there was a problem never took place.

What it cost

Two things, and both are real.

The DMV queries are not free. They are cheap when the server is idle and less cheap when it is already struggling, which is exactly when they run most often. Polling a suffering instance to ask why it is suffering adds to the suffering, in a small way that I have chosen to accept and would want to re-measure on a busier box than this one.

And the thresholds do not travel. "How many blocked sessions is too many" is not a general number, it is a fact about this workload on this schema with this batch job running at this hour. I tuned them by watching, which means they are correct here and worthless anywhere else. Anyone lifting this dashboard onto another estate is lifting the panels, not the numbers.

The green check was never the problem. The problem was that I had asked it a narrow question and then read its answer as a broad one.