Introducing Watchtower: Every Call. Scored.
Watchtower checks every AI-handled call against your own knowledge and rules and rolls the results into a single health score. Know your agent is performing, without reading a single transcript.
When you hand your phone lines to an AI agent, one question keeps coming back: how do I know it's actually doing a good job?
You could listen to recordings. You could read transcripts. You could spot-check a handful of calls and hope the rest are fine. That's what traditional QA looks like: expensive, slow, and incomplete.
Watchtower is the opposite.
What Watchtower Does
Update, September 2026. Scoring has changed since this post. Watchtower no longer rates calls on weighted 1-to-10 dimensions. Every call is now run through a fixed list of pass/fail checks, judged against the knowledge and rules your agent actually had on the call, plus any expectations you add in plain language. A fail has to quote the transcript, and a second pass verifies it before it counts. The score is the share of checks passed. The categories below survive as groupings of those checks; the weights and the 1-to-10 scales do not. Caller sentiment is still reported but is no longer part of the score. The current description is in the Watchtower help article.
Every call your agent handles is automatically scored on five quality dimensions the moment it ends. No delay, no sampling, no human reviewers. Every single call.
The five dimensions:
Resolution (30% weight): Did the caller get what they needed? If someone called to book an appointment, was it booked? If they had a billing question, was it answered? Resolution is the most heavily weighted dimension because it's the one that matters most to your callers.
Accuracy (25%): Was the information correct? This is where hallucination detection lives. If your knowledge base says you're open until 6 PM, did the agent say 6 PM, or did it guess 5? Accuracy scoring catches the moments when an AI sounds confident but says something wrong.
Professionalism (20%): Natural conversation flow, clear communication, appropriate greeting, smooth handoffs. This measures whether the call felt right. A caller who gets the right answer in the wrong tone is still a dissatisfied caller.
Sentiment (15%): How did the caller feel? Not at one moment, but across the arc of the conversation. Did they start frustrated and end satisfied? Did the agent make things better or worse? Sentiment tracks the emotional journey of every call.
Escalation Handling (10%): When a call needs a human, does the handoff happen correctly? When it doesn't need one, does the agent resolve it without unnecessary transfers? Both matter.
These five scores are combined into a single number from 0 to 100: your Agent Health Score.
The Health Score
Think of it like a credit score for your AI agent.
- 90-100 (Excellent): Your agent is performing at expert level. Most calls resolved cleanly, callers are happy, information is accurate.
- 75-89 (Good): Performing well with minor areas for improvement. This is where most well-configured agents land.
- 60-74 (Needs Attention): Consistent issues detected. Time to review flagged calls and update your knowledge base.
- Below 60 (Critical): Significant problems that need immediate attention. Watchtower will alert you before you get here.
The health score is a 30-day rolling average, weighted toward recent performance. If you have a bad week but fix the issues, your score recovers within days, not a month.
Zero Effort Monitoring
Here's the part that makes Watchtower different from "go check a dashboard every day":
Weekly digest emails. Every Monday morning, you get an email with your health score, dimension breakdown, flagged calls, and areas for improvement. If everything's fine, it takes 10 seconds to scan and move on. If something needs attention, you know exactly what.
Automatic alerts. If your health score drops below 70, you get an immediate alert, not Monday's summary. Same if your score declines for three consecutive periods.
Flagged calls. Watchtower automatically flags any call where a dimension scores 3 or below, the agent gave clearly wrong information, or a caller was left unresolved. These calls surface in your dashboard and weekly digest with specific rationales for why they were flagged.
You don't have to monitor anything. Watchtower watches, scores, and tells you when something matters.
How It Works (The Short Version)
There's no setup. There's no separate tool. There's no additional cost.
When a call ends, quality scoring happens automatically as part of the post-call analysis. Scores appear on your call history immediately.
The Watchtower dashboard in your account shows everything: health score with trend, dimension breakdown with averages, flagged calls with expandable rationales, and score distribution across all your calls.
Here's exactly what it looks like:
Watchtower
Every call checked against your knowledge and rules. Fails quote the transcript and are verified before they count.
Each bar is the share of checks passed in that category over 30 days. Caller mood averaged 8.4/10; it is shown for context and is not part of the score.
Weakest category: Escalation (9.0/10)
3 calls flagged in the last 30 days. Allison can read which checks failed, with the quoted lines, and tell you whether the fix is in your knowledge, your rules, or on our side.
Caller scheduled a cleaning for next Tuesday. Confirmed time and sent SMS reminder.
9 of 9 applicable checks passed.
- Understood what the caller wanted
Understood the request on the first turn.
- Need met, or clearly scoped with a next step
The caller left with what they came for.
- Captured what the flow required
Name, preferred day and mobile number captured and read back.
- No unsupported business facts
Every fact stated matches the configured knowledge.
- No self-contradiction
No conflicting statements.
- Honest when knowledge ran out
Every question was covered by the knowledge.
- Nothing a caller would complain about
Polite and unhurried throughout.
- Clear and on topic
No loops or holding lines.
- Ended the call cleanly
Confirmed the next step and said goodbye.
- Escalated when a rule matched
No escalation rule matched.
- No unnecessary escalation
No transfer happened.
- Explained the handoff
No transfer happened.
- Offer a bookingYour expectation
A booking was made.
- Never quote a priceYour expectation
Pricing did not come up.
Caller asked about Saturday hours. Agent provided correct hours and offered to book.
Caller asked if Delta Dental is accepted. Agent confirmed and explained coverage.
Caller rescheduled Thursday appointment to Friday. Agent confirmed the change.
Escalated when a rule matched
New patient called to register. Agent collected info and booked initial exam.
Caller asked about an outstanding balance. Agent provided the amount and explained payment options.
Post-procedure follow-up. Agent confirmed recovery instructions.
No unsupported business facts
No self-contradiction; Nothing a caller would complain about; Escalated when a rule matched
1–10 of 10
Your expectations
Add anything you want every call held to, in plain language. Scored like the core checks and counted in your score.
- Offer a booking
Every caller who asks about a service is offered an appointment before the call ends.
- Never quote a price
Do not quote treatment prices; explain that pricing depends on the exam and offer a consult.
Core checks
The same for every business. Switch one off if it does not apply to how you work; it stops counting from the next call.
- Understood what the caller wanted
Allison worked out what the caller was after instead of pursuing something they did not ask for.
- Need met, or clearly scoped with a next step
The caller got what they needed from your knowledge, or was told clearly what you do not offer and given a next step. Saying "we don't do that" when you don't is a pass. Leaving the caller waiting with no progress until they give up is not.
- Captured what the flow required
When a message, callback number, order or booking detail was needed, Allison captured it and the caller confirmed it.
- No unsupported business facts
Every hours, price, service, policy, address or staff claim came from your knowledge or a lookup result.
- No self-contradiction
Allison did not contradict herself, your knowledge, or a lookup result during the call.
- Honest when knowledge ran out
When your knowledge did not cover a question, Allison said so rather than guessing.
- Nothing a caller would complain about
No rudeness, dismissiveness, talking over the caller, or sustained confusion.
- Clear and on topic
No loops, no proceeding before the caller answered, no reading internal rules to the caller. The short hold messages Allison plays automatically while she works on an answer never count against her.
- Ended the call cleanly
When Allison ended the call, the next step was confirmed and she said goodbye. Callers hanging up is not counted against her.
- Escalated when a rule matched
When one of your escalation rules applied, Allison did what the rule says: transfer, take a message, or arrange a callback.
- No unnecessary escalation
Allison did not hand off something she could have handled from your knowledge. Transfers the caller asked for, or that only a person can do, are fine.
- Explained the handoff
When transferring or arranging a callback, Allison told the caller what would happen next.
Why 100%, Not 5%
Traditional call QA works by sampling. A quality team listens to a random selection of calls, usually somewhere between 2% and 10%, and extrapolates from there.
If 3% of your calls have an accuracy issue and you're only sampling 5%, you might catch one. Or none. And you won't find out until a customer complains.
Watchtower scores every call because it can. There's no human listener bottleneck, and scoring happens within seconds of the call ending, not days or weeks later.
This means patterns surface immediately. If a new FAQ is causing wrong answers, you'll see accuracy scores drop in the next digest. If your escalation rules are misfiring, the escalation dimension will flag it. You don't wait for complaints to tell you something's broken.
What This Means for You
If you're running an agent on Allison, Watchtower answers the trust question.
You don't have to wonder if calls are going well. You don't have to listen to recordings to find problems. You don't have to hire QA staff or build monitoring tools.
You get a score that tells you at a glance, and the dimension breakdown that tells you exactly where to look if something needs fixing.
Included on Every Plan
Watchtower is not a premium add-on. It's not a higher tier feature. Every Allison Voice plan (Starter, Professional, and Business) includes full Watchtower scoring, the dashboard, weekly digests, and alerts.
Quality monitoring is foundational, not optional. If you're trusting an AI with your phone calls, you should be able to see how it's performing. That shouldn't cost extra.
Watchtower is live for all Allison Voice customers. Explore how it works or call Allison to set up your agent and start seeing scores from day one.
Ready to build your agent?
Set up your AI voice agent in minutes. Start answering calls 24/7.