The scary thing about a broken AI workflow isnt that it errors. Its that it succeeds. It completes, logs a green tick, and moves on - while booking the wrong date, writing nonsense, or burning your budget because one expensive step ran twice. Most monitoring dashboards wont tell you any of that. These four numbers will.

Why the dashboard lies to you

A workflow that returns a 200 status and a valid-looking JSON blob has, technically, worked. The platform is happy. The logs are clean. Meanwhile the output is wrong, the cost just doubled, or the agent made eight tool calls for a job that used to take two.

The useful question isnt “did it run?” Its “what number would actually make you stop, fix, or ship?” There are four of them. Everything else is theatre.

1. Execution - did it run, and did anything spike?

This is the foundation. You need three things logged on every run:

  • Did it complete? Pass or fail, not just “no error thrown.”
  • How long did it take? A latency spike often means a tool call hung or a retry loop fired silently.
  • How many tool calls did it make? If a task that normally takes two calls suddenly takes eight, something has gone wrong upstream - even if the final answer looks fine.

Tool-call count is the canary most people ignore. Log it. Set a threshold. If your agent hits tool_calls > threshold on a routine job, that run needs a human eye before the output goes anywhere.

2. Quality - was the answer actually correct?

This is the hard one, because “valid-looking” and “correct” are not the same thing. A response can be grammatically perfect, structurally sound, and completely wrong.

You dont need a fancy eval framework to start. Pick one thing the output must contain or must not contain, and check it programmatically on every run. Examples:

  • A booking confirmation must include a date that falls in the future.
  • A product description must mention the product name.
  • A customer reply must not contain the word “unfortunately” if your brand voice forbids it.

A simple assertion like this catches the failure mode where a prompt or context change quietly breaks the output without triggering any error. Write it as a post-step check, not a manual review:

if not contains_future_date(output["confirmation_text"]):
    flag_for_review(run_id) # dont let it reach the customer

Start with one quality check. Add a second only when a real failure demands it.

3. Efficiency - what did this task actually cost?

Cost is the metric that bites you in silence. A prompt gets longer, a new context window gets added, and suddenly every run costs three times what it did last week. You wont notice until the invoice arrives.

Log token usage and, where your provider exposes it, a cost estimate on every run. Most APIs return this in the response body. For OpenAI-compatible endpoints it looks like this:

{
  "usage": {
    "prompt_tokens": 1240,
    "completion_tokens": 380,
    "total_tokens": 1620
  }
}
# multiply total_tokens by your per-token rate to get run cost

Track cost-per-task, not cost-per-month. Monthly totals hide the individual run that went rogue. If a single task’s cost jumps more than 50% after a prompt or context change, treat that as a breaking change and review it before it scales.

This is the lesson from a build we wrote about separately: expensive actions need idempotency, cost visibility, and a hard stop. Without those three things, one bad run can multiply itself before you realise whats happening. You can read the full account in the the £700 automation mistake article.

4. Safety - did it try something it shouldnt have?

Safety events are low-frequency but high-consequence. You need to log three things:

  • Guardrail hits - did the agent attempt an action your system is supposed to block?
  • Restricted-action attempts - did it try to write to a database, send an email, or call an endpoint it has no business touching?
  • Human escalations - did it correctly hand off when it was out of its depth, or did it guess and carry on?

A rising guardrail-hit rate after a model update or prompt change is a signal the agent’s behaviour has drifted. Catch it here, not in a customer complaint.

Where to start

Dont build a dashboard on day one. Pick one quality metric and one cost metric for your most-used workflow. Log them somewhere you actually look - a Slack message, a simple spreadsheet updated by a webhook, anything. Add execution and safety logging once you’ve seen what a real failure looks like in your numbers.

The whole point of these four metrics is that they answer the question a business owner actually cares about: if your best AI workflow went wrong today, would you know before your customer did?

If the honest answer is no, lets have a conversation about what monitoring looks like for your setup.