If you've tried to run a voice interface in Hebrew, you already know: the demos on TED-talk stages don't reflect what happens when you put a microphone in front of an Israeli clinician in a noisy hospital corridor. Hebrew's grammatical structure, its extensive code-switching with English, domain-specific clinical and operational vocabularies, and the accent range across Israeli dialects all conspire to break the consumer voice stack the moment it leaves the demo floor. Here's the production stack we actually ship - with the specific places it breaks and what costs more than anyone expects.

The fundamental problem: Hebrew is a minority language in a majority-English training pipeline

OpenAI Whisper was trained on something like 680,000 hours of audio. Hebrew represents a single-digit percentage of that. For consumer speech - podcasts, interviews, clean-room studio recordings - Whisper in Hebrew is acceptable. For production voice agents running in clinical, logistics, or call-center environments, acceptable is not good enough. You need fine-tuning, domain vocabulary, and often a different model for different stages of the pipeline.

The rule-of-thumb we use: whatever your English voice system costs, triple it for Hebrew production, and add another 40% if you need to handle bilingual code-switching.

Stage 1 - ASR: start with Whisper, fine-tune for your domain

OpenAI Whisper (and its Whisper-Turbo successor) is a reasonable Hebrew ASR baseline. It's not good on Hebrew out of the box - it's just the best general-purpose starting point. Where it degrades sharply:

• Clinical vocabulary. Drug names, anatomical terms, procedure codes, medication doses. Hebrew transliterations of English drug names have multiple spelling conventions (פנטניל / פנטאניל / fentanyl - same drug, three strings). Generic Whisper will pick one at random.
• Fast, colloquial speech. Israeli Hebrew is spoken fast, with syllable elision and heavy consonant clusters. Consumer ASR was trained on cleaner academic register.
• Background noise. Hospital corridors, warehouses, dispatch rooms, call centers. Consumer ASR expects a 3-foot distance to a clean mic. Your production environment does not provide that.
• Accents. Russian-Hebrew, Arabic-Hebrew, English-accented Hebrew, Mizrahi vs. Ashkenazi vowel quality. The accent range across Israel is substantial and Whisper's distribution-mean accent is none of them.

What works: fine-tune Whisper on customer-specific audio. We collect 10–50 hours of in-domain audio (clinical consultations, dispatch calls, call-center recordings), transcribe carefully, and fine-tune. Word error rate drops from ~25-35% on generic Whisper to 8-12% on our fine-tuned checkpoint for the specific customer. That delta is the difference between a voice agent that works and a voice agent that embarrasses.

Stage 2 - Hebrew NLP: use Dicta, not multilingual models pretending

Once you have a Hebrew transcript, everything downstream (entity extraction, summarization, intent classification, translation) was probably trained on a multilingual model where Hebrew was ~1% of the training data. You can feel this in production: entity recognition misses common Hebrew names; summarization defaults to Modern Standard Arabic-inflected phrasing; intent classification collapses semantic distinctions that matter clinically.

The Dicta research group at Bar-Ilan University has released Hebrew-specific language models - DictaLM, DictaBERT, and specialized models for historical Hebrew and modern Israeli text. For anything post-transcription, Dicta often outperforms a multilingual model at a fraction of the compute cost. We route Hebrew text through Dicta for entity extraction and classification; we use Bedrock's Claude Sonnet (or a bilingually-tuned variant) for summarization and generation.

Concrete example from a hospital deployment: named-entity recognition on Hebrew clinical notes. Generic multilingual XLM-RoBERTa: F1 of 0.68. Fine-tuned DictaBERT on our clinical corpus: F1 of 0.91. Same text. Same task. Different world.

Stage 3 - Code-switching is the hardest part

Israelis switch between Hebrew and English mid-sentence, often mid-clause, often mid-word. Clinical examples from our corpus:

"הפציינט עם ה-acute pancreatitis קיבל IV morphine, ונשלח ל-CT."

"הייתה לה tachycardia, פה 110, BP 90 על 60, ניתנה bolus ו-follow-up ב-ICU."

Half Hebrew, half English, completely normal Israeli clinical speech. The ASR has to transcribe both scripts correctly; the downstream NLP has to understand both and produce a coherent representation.

What works: we built a two-pass system that runs Whisper in Hebrew mode and Whisper in English mode on the same audio, then uses a classifier trained on code-switched corpora to decide which parts of the output to take from which pass. It's not elegant, but it beats any single-pass approach by 15+ percentage points on code-switched clinical audio. Downstream, Claude Sonnet handles the mixed-script text naturally - one of the few cases where a large multilingual LLM outperforms a Hebrew-specialized model, because code-switching is exactly the edge case multilingual models were trained on.

Stage 4 - Domain vocabulary will consume your budget

"פנטניל" vs "פנטאניל" vs "fentanyl" - same drug, three spellings. Multiply this by every drug, every medical condition, every instrument, every abbreviation. For Atlantic Logistics, the equivalent was shipping-document terminology: "POD" vs "proof of delivery" vs "דו״ח מסירה"; carrier names; SCAC codes; port codes.

What works: we build a customer-specific domain dictionary - typically 2,000–8,000 terms depending on vertical - with all acceptable spelling variants tagged. This dictionary feeds three systems: (1) a custom-trained Whisper fine-tune that preserves canonical spellings, (2) a post-ASR normalization step that maps variants to a canonical form, and (3) a Claude system prompt that includes the dictionary inline as context. This is unglamorous work and it's the work that makes the difference between a 20% WER voice agent and a 5% WER one.

Budget rule: plan 3–4 weeks of domain-vocabulary curation for any clinical or logistics voice deployment. It's always more than anyone estimates on day one.

Stage 5 - TTS is where brand is either made or lost

For Hebrew text-to-speech we default to Amazon Polly - it's available, it's acceptable, and it integrates with the rest of the AWS stack. When brand voice matters, we use OpenVoice or a similar brand-tunable system to generate a distinctive voice that matches the customer's identity. The difference is immediately audible. Generic Polly sounds like every corporate IVR you've ever hated; a tuned OpenVoice clone can sound warm, measured, and specific.

Two production notes: (1) synthesize early-morning and late-night variants if your agent runs 24/7 - the right softness at 3 AM is different from the right crispness at 9 AM; (2) test on the actual speaker hardware your users will hear the agent on. A Polly voice that sounds clean in studio often sounds thin on a call-center handset or an IoT speaker.

Stage 6 - Real-time latency: WebRTC, SFU, and careful buffer management

A voice agent is judged by perceived response time. >800 ms between utterance end and system response starts to feel slow; >1500 ms feels broken. That's a very tight budget to fit: ASR, NLP, LLM call, TTS, and network round-trip.

The production stack we run: WebRTC for the media channel; an SFU to handle multi-party and recording needs; sub-second ASR via a streaming Whisper instance; a hot Claude Sonnet connection on Bedrock (we pre-warm); streaming TTS that starts emitting audio as soon as the first sentence is ready. Measured end-to-end latency p50 on MAIA's clinical voice deployments runs 620–740 ms. P99 runs around 1.1 seconds. Anything higher and the clinicians disengage.

Stage 7 - SIP/PBX integration is where most voice projects die

The AI is the easy part. The hard part is integrating the voice agent into the customer's existing telephony stack: an on-prem PBX, a SIP trunk, a multi-region call center with regulatory recording requirements, and a dozen call flows with "warm transfer" and "context-preserving handoff" semantics.

We've shipped this at Atlantic Logistics (dispatcher voice agent integrated over SIP into the existing TMS phone tree) and at MAIA (bilingual clinical voice agent embedded inside hospital telephony). In both cases, the AI work took ~6 weeks and the telephony integration took another ~8–12 weeks. If your Voice AI project plan doesn't have at least 2 months for SIP/PBX integration work, the plan is wrong.

Watch for: the customer's PBX vendor may need to open a support ticket to enable SIP features; some carriers charge per-channel for media; recording retention policies vary by region; and context-preserving transfer (passing agent state to a human operator) is rarely a one-API-call operation - it's usually a bespoke integration.

What it costs, honestly

A production bilingual Hebrew-English voice deployment, for a mid-market enterprise or a government agency, lands in the following range:

• 6–10 weeks for a first voice-agent workflow, assuming the integration target (PBX/SIP, TMS, EMR) is accessible and the compliance posture is defined.
• 3–4 weeks of domain-vocabulary curation and fine-tuning.
• 8–12 weeks of telephony integration, in parallel with the AI build.
• Ongoing: monitoring, retraining (typically quarterly), prompt evolution, and voice-of-customer feedback loops.

You can go faster with fewer compliance constraints and a cleaner integration target. You cannot usually go faster than 6 weeks and still ship something that will survive Monday morning.

When not to build voice AI

If your use case can tolerate a text interface and your users can type, use text. Voice adds latency, cost, integration complexity, and accent-and-noise robustness requirements that text does not. We've talked several prospects out of voice projects because the underlying workflow was better served by a chat assistant. The voice ROI is strong when: the user can't type (clinician with gloves, dispatcher on the phone), voice is the cultural default (call centers, hospital intake), or the interaction is multi-party and conversational (meeting transcription, field-service triage).

If you're shipping a bilingual voice product in Israel - or an English voice product that will also need to handle Hebrew speakers - let's talk. We'll bring the architecture diagram and the specific places we've seen the stack fail, not a deck.