How to Test Voice AI for Consumer-Brand Phone Support: A Pilot Scorecard

TL;DR

• Use this buyer-run protocol to compare voice AI pilots, not to rank vendors. Set pass/fail thresholds against your brand’s phone-support baseline rather than a borrowed benchmark.
• Run repeatable calls for order status, returns, after-hours coverage, caller verification, and seasonal spikes. Include unavailable data and failed transfers.
• Score each vendor on verified resolution, transfer quality, containment, latency, failure handling, and cost per resolved call. Set the weights before testing. Check outcomes against your own order records and call reviews, not just vendor dashboards.
• Confirm that the agent works with your phone and CRM systems, then apply separate go/no-go gates to each workflow. An order-status pass does not establish that returns will pass.

Why a scripted demo won't tell you how a voice AI agent handles your call volume

A scripted demo shows how a voice AI agent handles a caller and an order the vendor has chosen. Your customers may give partial order numbers, change their request mid-call, or call when order data is unavailable. A smooth demo cannot show whether the agent verifies identity before sharing details, acknowledges missing information, or transfers the caller with useful context.

Vendor-reported resolution rates may count a call as successful because no transfer occurred, even if the caller received a wrong answer or called again. Before seasonal demand rises, run a pilot against your own call mix, including after-hours calls and periods of higher volume. Check recorded answers against your order records, return policies, and subsequent customer contacts. Those buyer-controlled records let you distinguish a resolved call from one the agent merely ended.

Higher call volume also tests the path around the agent. Calls must reach it through your phone system, and transfers must still connect when human queues are busy. A pilot should record those outcomes alongside answer accuracy rather than rely on a vendor dashboard alone.

What to test: the seven workflows that matter for phone-support volume

Test your highest-volume call types and the capabilities needed to handle them safely. Use known outcomes from your own records, including cases the agent should refuse or transfer. The seven areas below include both caller requests and operational checks.

1. Order-status lookups. Ask about orders that have shipped, are delayed, or cannot be found. Check the answer against the order record, and watch whether the agent admits when current data is unavailable.

2. Returns within policy. Use eligible purchases with different return windows and conditions. Check whether the agent applies the correct policy to the specific purchase rather than giving a generic return answer.

3. After-hours coverage. Call when your staff are offline. Confirm that the agent can resolve supported requests and give callers a clear next step when a person must follow up.

4. Seasonal spikes. Run concurrent calls across common and difficult scenarios. Measure whether callers reach the agent, receive timely answers, and can still transfer when call volume rises.

5. Telephony and CRM fit. Place calls through your existing phone setup and check whether the agent reads and records the right information in the permitted systems. A vendor's integration claim does not establish that your configuration works.

6. Caller verification. Test callers who provide correct, incomplete, and conflicting details. The agent should withhold private order information until it can verify the caller under your rules.

7. Human transfer with context. Send complex calls to staff and inspect what arrives with each call. The receiving staff member should receive the caller’s request and reason for transfer, along with verified details they are authorized to see, so the caller does not have to start over.

Treat live order and inventory answers as conditional on authorized data access and freshness checks. For inventory questions, verify access to the correct location's records. If the agent cannot trust the data, it should say so and route the caller to a person.

Building reproducible test scenarios for each workflow

Build each test call from a case record that another person can replay. Record the caller identity the agent should accept, the relevant order and CRM facts, the applicable policy version, and the expected outcome. Save a dated copy of those records. Your QA reviewer can then compare the recording with the facts available at call time, rather than relying on the agent’s “resolved” label.

For each workflow, prepare a routine call, an edge case, and a repeat test for a known failure or rule change. General AI-agent testing guidance can help structure cases, but phone tests must also account for speech and call conditions. Replay the same case with an interruption or background noise while keeping the order facts and expected answer fixed. Record the call route and agent version so another reviewer can reproduce the result. Add a call in which someone asks for another customer’s order details.

Give every workflow a verifiable outcome.

• For order status, seed an order with a known status and check the answer against the order record.
• For returns, use a purchase with a known date and policy outcome, then check whether the agent applies the right policy version.
• For after-hours coverage, call outside staffed hours and check the response against your published schedule and routing rules.
• For seasonal spikes, repeat the same cases at concurrent call volumes based on your past peaks. Record delays and dropped calls.
• For telephony and CRM fit, place calls through the planned phone route and check whether authorized records appear or update as expected.
• For caller verification, test both a matching identity and one that cannot be confirmed. Record what information the agent discloses.
• For human transfer, check whether the call reaches the intended queue and whether the staff member receives the relevant conversation context.

Size the set around your call mix. Give high-volume workflows more routine cases, but keep separate edge and repeat cases for every workflow you plan to launch. Record the agent configuration and test date with each result so you can run the same calls after a change and compare like with like.

Failure cases to script deliberately

Script failure calls with a known order record, the current return policy, and an expected recovery action. Run the same scenarios for each vendor. Have your own QA reviewer score what the caller hears and what reaches a human agent, rather than relying on the vendor’s call outcome label.

• Unavailable order data. Disable access to a test order during the call. The agent should say it cannot confirm the status and offer an appropriate handoff. Score a guessed delivery date as a failure.
• Stale order data. Give the agent an older status while your test record shows a newer one. Check whether the agent uses an available timestamp or freshness check to identify the older status. If it cannot establish which status is current, it should say so rather than guess. The agent should not present an unverified status as current.
• Ambiguous caller identity. Use two test customers with similar identifying details, then have the caller provide incomplete or conflicting information. The agent should request the required verification or transfer the call without revealing either customer’s order details.
• Out-of-policy return. Request a return for a test order outside your stated window. The agent should explain the applicable limit and offer escalation if your policy allows it. An invented exception or return label fails the case.
• Order or CRM connection failure. Interrupt a lookup or return-record update after the caller makes a request. The agent should distinguish a completed action from an attempted one and pass the failure reason to staff if it transfers.
• Transfer failure. Make the human queue unavailable or interrupt the handoff. Check whether the agent follows your approved fallback instead of leaving the caller in a loop. When a transfer succeeds, check that staff receive the relevant conversation context.

Score these cases separately. A policy-required transfer can pass even though the agent did not resolve the call alone. Guessing, disclosing unverified customer data, or claiming an action succeeded when the connection failed should not pass.

The weighted pilot scorecard

Use the same scorecard for every vendor, but set weights and scoring thresholds around your own call mix. Score each workflow separately so strong order-status results cannot hide unsafe returns handling. A QA reviewer should check call recordings and CRM records against the expected outcome for each scripted scenario.

Set the weights to total 100%. Before testing, document a 0-to-5 scale for each criterion. For accuracy and transfer quality, map observed rates to scores using your baseline and acceptable range. For latency and cost, reverse the scale so lower values earn higher scores. Assign 0 when the required evidence is missing, but report unverified calls separately rather than treating missing evidence as a measured failure. Specify whether you will score the worst workflow result, each workflow separately, or an aggregate. Multiply each score by its percentage weight, then divide the sum by 100 to obtain a total out of 5. Keep the underlying measurements beside the weighted score. Two vendors can earn the same total while failing in different ways, and a serious privacy or policy error may warrant a separate pass/fail gate.

Do not treat a call ending as proof of resolution. Intercom’s Fin documentation separates confirmed from assumed resolutions, a distinction worth applying to phone calls. Record explicit caller confirmation separately from calls that simply end without a transfer. Then check both against order records or other ground truth. Track same-issue recontact within 24, 48, and 72 hours as a signal of possible false containment, not conclusive proof that the first call failed.

Setting go/no-go thresholds with your own data

Before testing, use your own call records to set a baseline for each workflow. Compare order-status calls with order-status calls and returns with returns. Record the human-agent resolution rate, average handle time, and cost per correctly resolved call. Include comparable telephony charges and staff time in both baseline and pilot costs, and document any AI-specific charges separately. For after-hours coverage, use your current after-hours service as the comparison, since daytime staffing may offer a different level of service.

Set pass/fail gates for each workflow before you review pilot results. You may require stricter answer accuracy for returns, where a wrong policy decision can affect a refund, than for order-status lookups. Decide separately which failures require a pause, such as disclosing an order to an unverified caller or losing a caller during transfer. A high overall score should not cancel a failed safety gate.

Use the same cost definition on both sides of the comparison. Count AI charges and any human follow-up when you calculate cost per correctly resolved call, rather than counting every call the AI handled as resolved. Then compare the pilot with your baseline and check whether the results meet each workflow’s agreed gates. If the test set contains too few examples of a failure case to judge it, run more calls before making a go/no-go decision.

Verifying telephony, CRM, and order-system fit before you trust the pilot's results

An order-status score means little until you confirm which records the voice agent can read. Ask each vendor to identify the order system, the permitted API actions, and whether test calls use sandbox records or live data. Seed test orders with known answers, then check the agent’s responses against those records. If access fails or a record looks stale, score whether the agent says it cannot confirm the status and offers a human transfer.

Test the phone and CRM connections on the setup you would use in production. Place calls through your existing phone system, verify that the intended CRM record opens, and check what the receiving staff member actually sees after a transfer. A vendor’s ability to send a conversation summary does not establish that your agent desktop displays it. The order-status and returns guide covers verification and handoff mechanics in more detail.

SigmaMind AI is an option to evaluate if your brand has substantial inbound phone volume and existing telephony and CRM systems. Its documented capabilities include configurable conversational workflows, monitoring, and transfers that preserve conversation context. Ask SigmaMind AI to document the proposed phone and CRM connections, then verify those connections and the handoff context in your pilot. Its retail platform comparison provides vendor-selection context, but an order-system connection still needs confirmation for your retailer. For Shopify or Gorgias, request documentation for the specific connection a vendor proposes and verify its permitted actions in the pilot. Do not assume either is a native integration. Record documented capabilities separately from connections configured and observed during your pilot.

Running the pilot: sequencing, sample size, and who scores it

Set a pilot window with dates for scripted testing, a live-shadow or limited-production phase, and a final scoring review. Run scripted calls first so you can check answers against known order records and return policies. Include normal requests and deliberate failures, such as unavailable order data or a caller who cannot verify their identity. Move to live shadowing only after scripted calls meet the gates for each workflow. In shadow mode, compare the agent's responses with real calls without letting it answer callers or initiate transfers. Before routing a small share of real calls to the agent in limited production, confirm privacy controls and a human fallback. Use that phase to observe transfers and their effect on staff.

Size the test set around your call mix, not a universal minimum. Use recent phone records to estimate how often each workflow occurs. Before testing, specify how many calls you will review per workflow and failure case, and who will decide whether the evidence is sufficient for a go/no-go decision. Add scripted cases for rare failures that may not appear during the pilot window. If seasonal demand changes the mix or volume, test those conditions rather than assuming an ordinary week represents the peak.

Have your own QA listener score recordings against the pilot rubric and check factual answers against order and CRM records. Ask receiving staff whether transfers included enough conversation context to continue without making callers repeat themselves. Keep the vendor’s dashboard as supporting evidence, not the final score. Fix a re-test cadence before seasonal demand rises, and rerun affected scenarios after every prompt or model change so a passing score still applies to the version you plan to use.

Common scoring mistakes that let a weak pilot pass

A blended containment rate can hide calls that ended without a resolution. If a dashboard records only whether a transfer occurred, it may count a caller who hangs up after receiving stale order information as contained. Separate verified resolutions from abandoned and unresolved calls, then check recordings against order records. For phone calls, verify outcomes against your records rather than treating an ended call as resolved.

An intentional transfer should not count as an agent failure when your policy requires a person to handle the request. For an out-of-policy return, score whether the agent recognized the limit, reached the right person, and passed along the caller’s request. Score a failed connection or missing conversation context separately. A low transfer rate can look good while concealing calls the agent should have escalated.

Vendor dashboards cannot independently verify their own success labels. Have your QA reviewer compare a sample of recordings with order and CRM records, and check whether callers contacted support again about the same issue. If the vendor cannot provide enough evidence to verify a call’s outcome, mark it unverified rather than successful.

FAQs

How long should a pilot run before you decide?
Run the pilot until you have enough scored calls in each workflow to compare results with the thresholds you set in advance. Allow time to check whether callers contact support again about an issue the agent marked resolved. A calendar deadline alone cannot tell you whether the pilot passed.

How can you test seasonal spikes before peak traffic arrives?
Use simulated concurrent calls that reflect your expected mix of order-status questions, returns, and transfers. Increase call volume in steps, then compare answer accuracy, response time, dropped calls, and transfer completion against your buyer-set gates. Include after-hours tests when fewer human agents are available.

What if a vendor resists access to ground-truth QA?
Ask for the call recordings or transcripts, event logs, and dispositions you need to compare answers with your own order and CRM records, subject to your privacy controls. If you cannot independently check what callers heard and what happened afterward, treat the outcome as unverified rather than accepting the vendor’s resolution figure.

Can an order-status pass cover returns too?
No. A correct order lookup does not show whether the agent applies your return policy correctly or recognizes an exception. Score returns separately, including in-policy requests, out-of-policy requests, and cases that require a human decision.

Conclusion

Use the workflow gates and independently checked outcomes to decide which calls, if any, the agent can handle during seasonal demand. Keep a human route for cases where identity, order data, policy, or a system connection cannot be confirmed.

Retain the test calls, expected answers, scoring rules, and thresholds so you can apply the same gates when you compare another vendor or change a model or call workflow.