Field Notes From Building a Voice Agent for Arabic Callers

Field Notes From Building a Voice Agent for Arabic Callers

Notes from building, not from a case study

This is a different kind of piece from the others in this section. It is not analysis of someone else’s deployment. It is what we have learned building two customer-facing agents — Cue for chat and Ace for voice — and it includes the things that did not work.

We are writing it because the market is thick with capability claims and thin with engineering detail, and because the decisions below are the ones that determine whether a voice agent survives contact with real callers. They are not proprietary. Any team building in this space will hit them.


1. The hard part of Arabic is not Arabic

Every vendor in the region says it supports Arabic. On a written Modern Standard Arabic sentence, most now do — the frontier models have made general Arabic close to a solved problem, and the published benchmarks say so.

That is not what arrives on a phone line in Dubai.

What arrives is a caller switching between Arabic and English inside a single sentence, in Emirati, Saudi, Egyptian or Levantine dialect, over a compressed telephony codec, often with background noise, using a proper noun the system has never seen. Every one of those is a separate engineering problem and they compound.

Three things follow from that.

Code-switching is the normal case, not the edge case. We designed for it as the default rather than adding it later. A system that handles two clean monolingual paths and falls over on a mixed sentence will fail on a large fraction of real traffic in this market.

Recognition and output are different problems and deserve different answers. On the way in you need dialect recognition across at least the four major Gulf and Levantine variants. On the way out, the right voice depends on the institution: Gulf dialect reads as warm and appropriate for consumer services, and wrong for a bank or a ministry, where Modern Standard Arabic is what people expect. We made that a per-deployment choice rather than a product default.

Custom vocabulary is not a nice-to-have. Brand names, product names, branch names, Arabic personal names transliterated inconsistently across a client’s own systems. This is unglamorous configuration work and it is a large share of the difference between a demo and a deployment.

The honest summary of where we are: two languages, with dialect handling on the way in. Not all languages. We would rather state the bound than let a buyer discover it.


2. Latency is a product constraint, not a metric

On a phone call, silence is failure. A caller who waits two seconds for a response assumes the line has dropped or the system has broken, and starts talking over it — which corrupts the turn and compounds the delay.

We hold Ace to under 800 milliseconds to first response, measured end to end in our own environment. That is an internal engineering measurement, not a customer-verified figure from production traffic, and we say so because the distinction matters when you are comparing vendors.

What it forces architecturally is more interesting than the number. Speech recognition has to stream rather than wait for an utterance to end. The reasoning step gets a strict budget, which constrains how much retrieval and how many tool calls can happen inside a turn. Speech synthesis has to begin before the full response is composed. And if a function call into a CRM is slow, the agent has to say something while it waits, in the way a human would.

Any one of those decisions taken late requires rebuilding the others. Latency is not something you optimise at the end.


3. Model choice should be a configuration, not a foundation

The model layer is moving faster than any procurement cycle. A system architected around one provider’s API surface, prompt idiosyncrasies and function-calling behaviour is a system that has to be rebuilt when a better or cheaper option appears — which, on current evidence, is roughly every six months.

Ace runs on Gemini, OpenAI or Anthropic, chosen per workload and swappable without rebuilding the agent. This is not vendor neutrality as a marketing position. It is a hedge against a market where enterprise API spend shifted substantially between providers in a single year.

The practical requirement is an abstraction over model calls that is thin enough not to lose provider-specific capability and thick enough that swapping is a configuration change. Getting that boundary right is genuinely difficult, and it is worth doing.


4. Deflection is the wrong objective

Contact centre programmes are still routinely measured on deflection — the proportion of contacts that never reach a human. It is the wrong target, and it produces systems people hate.

A caller whose problem is not solved does not go away. They call back, they escalate, they complain on a channel you do not control. The deflection number improves and the cost does not, because the work reappears further down the line in a more expensive form.

The measure that matters is resolution: the contact is finished, and the customer does not come back about the same thing. That changes what the agent has to be able to do. It cannot only answer — it has to act, which means function calls into the CRM, the booking system and the ticketing system while the caller is still on the line. An agent that can describe a process but cannot execute it is a more articulate IVR.

It also changes what a good handover looks like. When the agent cannot resolve something, the human should receive the full context — what was asked, what was tried, what the system already knows about the caller — rather than a transfer that makes the customer start again. Most of the frustration people attribute to AI in customer service is actually a handover design failure.


5. Know what the system must not do

The CBUAE’s 2026 guidance describes three tiers of oversight — human-in-the-loop, human-on-the-loop, human-out-of-the-loop — and confines full autonomy to low-risk processes. That framing is useful well beyond financial services, and it maps directly onto how a voice agent should be scoped.

Our default boundaries: the agent can retrieve, confirm, book, reschedule and follow up autonomously. It escalates on anything that moves money outside a pre-authorised flow, anything involving a complaint or a vulnerable caller, anything where confidence in the recognised intent is low, and anything the client has designated as requiring a human regardless of confidence.

Deloitte’s 2026 research found only about a fifth of organisations have mature governance for agentic AI, while three quarters expect meaningful agent deployment by 2027. The gap is usually not a missing policy document. It is that nobody wrote down, before launch, which decisions the system may take alone.


6. What we have not proven yet

We would rather say this than have a buyer discover it.

Ace is built — inbound and outbound agent, CRM integration, working demo environment — and it has no production deployments. Our next milestone is a paid pilot on a live number, on a single high-volume call type, and after that a reference customer with published call-resolution and containment metrics from real traffic. Those metrics do not exist today and we will not imply otherwise.

Some things are honestly not there. Courtesy callback and virtual hold are not built. SMS and WhatsApp are not integrated. Beyond the CRM connector that is running, other integrations are connector work we have scoped but not shipped. And a demo we ran earlier this year broke in three places — which is a normal thing to happen and an abnormal thing to publish, but the alternative is selling roadmap in the present tense, and experienced operators can smell that immediately.

RAIL, our finance layer, is further along: one multi-brand UAE estate of around eighty outlets across two brands, deployed in thirty days, ingesting eight scheduled reports daily into four dashboards, currently reconciling against that finance team’s existing pack ahead of sign-off. That is a real deployment at a real stage, and the stage is user acceptance testing.


Why publish this

Because the standard we ask of platform vendors — Arabic tested rather than claimed, a contractual training boundary, deployable where the regulation requires, pricing that survives a real deal — is not a standard we can apply to others and exempt ourselves from.

And because a buyer choosing between agent platforms is better served by a vendor who states the bound than one who says yes to everything. The second kind is easier to buy from and much harder to deploy.

Leave a Reply

Your email address will not be published. Required fields are marked *