Inbound voice AI that hands off to a human with the context attached

A voice assistant that answers inbound calls, books appointments against live availability and updates the CRM, then transfers to a human with the full transcript and the caller's details already on screen. It listens for the moment a conversation stops going well — falling intent confidence, a shift in tone — and hands over at that point rather than after the caller has asked twice. The audio streams both ways over a WebSocket, so the assistant starts answering while the caller is still finishing the sentence.

A multi-site, appointment-based service business

The client

A multi-site, appointment-based service business

Industry
Enterprise customer support
Size
Enterprise

Challenge

Peak-hour callers were hanging up before anyone answered

The client's support centre was losing 35% of inbound calls during peak hours due to limited human agent availability. A 24/7 staffing model was financially unsustainable, and agent fatigue caused inconsistent data entry in their CRM — resulting in duplicate records and missed follow-ups that cost an estimated $2.1M annually in lost revenue.

For an enterprise processing thousands of appointment-based inquiries daily, every abandoned call represents lost revenue. The client needed more than a basic IVR menu tree — they required a natural-sounding AI capable of understanding caller intent, checking real-time availability across 200+ service locations, and recognising precisely when a human agent was needed to close high-value leads.

Solution

A streaming-first pipeline, so the pause before a reply stops being the product

We deployed a modular Conversational AI pipeline built on a 'Streaming-First' architecture. The system chains ultra-fast Speech-to-Text (STT) for real-time transcription, a fine-tuned LLM with RAG for intent recognition and tool-calling, and high-fidelity Text-to-Speech (TTS) — all connected via WebSocket-based audio streaming to bypass traditional request-response overhead.

To hit the sub-2-second latency target, we eliminated HTTP polling entirely in favour of full-duplex WebSocket connections. The AI was integrated directly with the client's CRM and scheduling APIs, enabling live availability lookups and appointment bookings without human intervention — reducing average handle time from 8 minutes to under 60 seconds for routine queries.

Calls we would still defend

  • Hybrid Semantic Routing — Built a real-time decision engine monitoring sentiment drift and intent confidence scores to trigger human handoffs before customer frustration peaks — not after.

  • Contextual State Transfer — Developed proprietary middleware that passes full transcripts and extracted structured data (caller name, ID, issue category, sentiment score) to the agent dashboard during transfer — eliminating the 'please repeat yourself' problem.

  • Noise-Resistant STT Pipeline — Fine-tuned speech recognition models on 50,000+ hours of mobile call audio to filter background noise common in real-world calling environments, improving transcription accuracy by 23%.

Results

What changed for the person who called

Before

Human agents overwhelmed by routine FAQs. 8-minute average wait times. Zero support coverage between 8 PM and 8 AM. 35% call abandonment rate during peak hours.

After

Instant 24/7 response across all time zones. Routine queries resolved in under 60 seconds. Human agents focused exclusively on complex, high-priority escalations. Call abandonment dropped to under 3%.

2s

Call Response Latency

Inbound calls answered in about two seconds, not after a queue

80%

Automated Resolution Rate

Most calls handled end to end; the rest passed to a person with context

4K+

Monthly Appointments Booked

Appointments written straight into the calendar each month

FAQ

The questions this one raises

What was actually built, the constraints it had to meet, and what we would do differently.

  • When the AI detects a complex issue or negative sentiment drift, it initiates a SIP transfer to the next available human agent. Simultaneously, the agent's screen displays a real-time summary including the full transcript, extracted entities (caller name, issue category, account ID), and sentiment score — so the customer never has to repeat themselves.

  • Yes. The system uses multilingual STT models trained on 50,000+ hours of diverse audio data, supporting over 20 languages. Regional accent handling is achieved through dialect-specific fine-tuning, maintaining 95%+ transcription accuracy across English, Hindi, Spanish, and Arabic variants.

  • The production system achieves 1.8-second average end-to-end latency — from the moment a caller finishes speaking to when they hear the AI response. This is achieved through WebSocket-based streaming, edge-deployed inference, and pre-computed response caching for high-frequency queries.

  • Audio is encrypted in transit with TLS 1.3 and at rest with AES-256, and identifying details are redacted from stored transcripts. Where data residency is a requirement, processing can be pinned to a single region. Those are mechanisms we built rather than certifications — THWorks holds no SOC 2 or HIPAA attestation, and neither does this system.

  • The AI connects via REST APIs and webhook integrations to popular CRMs (Salesforce, HubSpot, Zoho) and scheduling platforms. During calls, it performs real-time availability lookups, creates appointments, and updates contact records — all without human intervention. Proprietary systems are reachable the same way, through a custom integration.

Keep reading

Other builds, taken apart the same way

Want to talk about yours?