Implementing Real-Time AI in Enterprise Contact Centres: Architecture, Challenges, and Measurable Outcomes

Section 1

Introduction: The Gap Between Pilot and Production

The global call centre AI market was valued at USD 3.23 billion in 2024 and is forecast to reach approximately USD 25.84 billion by 2034 — a compound annual growth rate that reflects genuine industry transformation, not just hype. Yet most organisations with "AI contact centre initiatives" have something more modest: a chatbot bolted onto an IVR, or a sentiment dashboard that nobody checks in real time.

The gap between a compelling pilot and a production-grade system is where most projects silently die. In regulated environments — pension funds, banks, insurers — that gap is wider because the stakes of getting it wrong are higher. A member calling about their retirement benefits is not an ideal subject for an experimental AI system.

What follows is a technical account of how we closed that gap at a major Canadian pension fund: what we built, why we made the architectural choices we did, what failed before it worked, and what the numbers looked like twelve months later. The intent is to give other practitioners a reference point that goes beyond the vendor marketing deck.

Section 2

Problem Statement: What the Existing System Could Not Do

The organisation operated a high-volume contact centre handling pension enquiries: account balances, contribution statements, retirement projections, beneficiary updates, and escalations. The existing IVR was a conventional DTMF (touch-tone) menu tree with no natural language capability. Agents had no real-time support — no automated summaries, no next-best-action prompts, and no structured post-call data capture beyond manual notes.

Key pain points identified during discovery:

After-call work (ACW) accounted for 20–25% of total handle time. Agents spent significant time writing call notes manually after each interaction. Research confirms this is an industry-wide problem — ACW typically represents 20–30% of handle time, and AI automation of summaries can reduce it by up to 50%.

No real-time quality assurance. Supervisors could only review recorded calls after the fact — typically a sample of less than 5% of total call volume. Compliance breaches or policy deviations were discovered days later, not during the call.

High call volumes for queries resolvable without a live agent. A significant proportion of inbound volume was simple account status or document request queries — tasks that, with the right self-service capability, could be handled without agent involvement.

No structured intent data. Every call was an unstructured black box. The organisation could not answer basic operational questions like: what are members most commonly calling about this month? Which topics generate the most repeat calls?

Section 3

Technical Architecture

The core design objective was a low-latency, streaming AI pipeline that could process call audio in near real-time — delivering insights to agents during the call, not after it. The architecture has five primary layers.

3.1 Audio Ingestion: Five9 Voicestream

Five9 Voicestream is a WebSocket-based real-time audio streaming API that allows third-party applications to receive a live duplex audio stream from an active Five9 call. Each call produces two separate audio channels — agent and member — as PCM audio at 8kHz/16-bit, interleaved in 20ms frames.

Voicestream was selected over batch-processing alternatives (post-call transcription) for one critical reason: real-time agent assist requires latency under 2 seconds end-to-end. Post-call processing was useful for analytics and QA but could not support the live coaching and summary-in-progress use cases.

3.2 Transport Layer: gRPC Streaming

Audio frames received from Voicestream are forwarded to the processing backend via bidirectional gRPC streaming. The choice of gRPC over REST was deliberate: HTTP/1.1 REST introduces per-request connection overhead and does not support genuine server-push streaming. gRPC over HTTP/2 provides multiplexed streams over a single persistent connection, with binary protobuf serialisation that is significantly more compact than JSON.

The service definition for audio streaming:

This bidirectional contract — audio frames in, NLP events out — allowed us to develop and test each NLP stage independently, and swap components without touching the transport layer.

3.3 ASR: Automatic Speech Recognition

Raw PCM audio is first passed to an ASR model producing partial (streaming) transcripts. We evaluated three ASR options:

Option	Latency (P95)	Accuracy (Financial domain)	On-prem option	Decision
Azure Speech (streaming)	380ms	High — custom model fine-tuned	No	Selected
Google Speech-to-Text	420ms	High	No	Deselected
Whisper (self-hosted)	1,100ms (GPU)	High	Yes	Deselected — latency

We fine-tuned the Azure Speech custom model on a corpus of anonymised call transcripts from the previous 18 months — approximately 8,000 hours of audio. The financial domain vocabulary (pension terminology, product names, regulatory terms) reduced word error rate by 22% compared to the base model.

ASR produces partial transcripts — incremental word sequences before the speaker finishes a sentence — and final transcripts when end-of-utterance is detected. The NLP pipeline processes both, but downstream models act primarily on final transcript segments to reduce noise.

3.4 NLP Pipeline: Four Concurrent Stages

Final transcripts are fanned out to four parallel NLP processors. Fan-out rather than sequential processing was a key latency optimisation — the total pipeline latency is bounded by the slowest single stage, not the sum of all stages.

Stage 1 — Real-Time Sentiment Analysis

A fine-tuned BERT-based classifier assigns a sentiment score (positive / neutral / negative / escalation-risk) to each transcript segment on a rolling 30-second window. The model was fine-tuned on labelled call transcripts annotated by QA supervisors. Escalation-risk is a fourth label beyond standard three-class sentiment — it fires when transcript content matches patterns associated with member complaints that historically escalate to formal complaints or regulatory referrals.

Sentiment scores are pushed to the agent desktop in real time. The intent is not to distract agents with a colour-coded dashboard, but to surface a discreet indicator when the sentiment trend is deteriorating — giving agents an earlier signal to change approach.

Stage 2 — Intent Classification

A multi-label intent classifier based on the DIET (Dual Intent and Entity Transformer) architecture assigns one or more intent labels to each utterance. The intent taxonomy was developed collaboratively with contact centre operations: 34 top-level intents covering the full range of member query types, with a confidence threshold of 0.72 before any intent is committed to the output stream.

Intent data serves two purposes: real-time next-best-action recommendations displayed to agents, and aggregate intent analytics for operational reporting. Intent classification also feeds the summariser — knowing the primary intent of a call allows the summary model to use the correct template structure.

Stage 3 — Topic Extraction

A separate extraction layer identifies named entities and key topics — account numbers (pattern-matched, not stored), product types, dates, contribution periods, benefit types. This differs from intent classification: intent answers "what does the member want?", while entity/topic extraction answers "what are the specific things they are talking about?"

Topics are used primarily for post-call data enrichment and search — allowing supervisors to query "all calls mentioning early retirement" across a date range without listening to recordings.

Stage 4 — Automated Call Summarisation

A generative summarisation model produces a structured call summary — call reason, key discussion points, actions committed to by the agent, and next steps — in real time as the call progresses, with a final summary generated within 8 seconds of call disconnection. The summary is automatically written to the CRM record, eliminating the manual ACW step for the majority of calls.

We used a fine-tuned GPT-based model (Azure OpenAI Service) with a structured output prompt enforcing a JSON schema. The structured output constraint was critical for CRM integration — free-form text summaries cannot be reliably parsed into CRM fields.

Section 4

Compliance and Data Architecture

Deploying a real-time AI system that processes voice data from pension fund members in Canada requires navigating several regulatory and governance constraints. This section describes the key constraints and how the architecture addresses them.

4.1 Data Residency

All audio processing and NLP inference runs in Canada-region Azure infrastructure. Raw audio frames are not persisted outside the call session — they are processed in-flight and discarded. Only transcripts, sentiment scores, intent labels, and summaries are retained, in encrypted form, in the data platform.

4.2 PII Handling

A PII redaction step runs immediately after ASR transcription. Account numbers, SIN fragments, dates of birth, and other identifiable patterns are replaced with typed placeholders ([ACCOUNT_NUMBER], [DATE_OF_BIRTH]) before any NLP model processes the transcript. This ensures no personally identifiable information is ever sent to external model endpoints.

4.3 Consent and Member Disclosure

Members are notified at the start of each call that the interaction is AI-assisted. The IVR opening message was updated to include explicit disclosure language approved by the compliance and privacy teams. Opt-out capability was implemented (though take-up rates were negligible in practice).

4.4 Model Governance

All models were registered in the organisation's AI model registry with full documentation of training data, evaluation metrics, and bias assessments. Quarterly model performance reviews were built into the operational cadence. Model updates require a formal change management process including QA sign-off and compliance review — the same process applied to changes in any other production financial system.

Section 5

Implementation Phases

The deployment was structured across four phases over approximately nine months, each delivering incremental value while managing risk.

Phase 1 — Streaming Infrastructure and Transcription (Months 1–3)

Establishing the Five9 Voicestream integration, gRPC transport layer, and ASR pipeline without any NLP. The deliverable was a reliable real-time transcription service validated at production call volumes. Success criteria: P95 transcript latency under 600ms, word error rate under 12% on financial domain vocabulary.

Phase 2 — Sentiment and Agent Widget (Months 4–5)

Deploying the sentiment classifier and the agent desktop widget — a lightweight browser extension overlaid on the existing agent desktop. Phase 2 was deliberately limited to passive display: agents could see sentiment trends but the system made no automated recommendations. This phase was primarily about building agent trust in the AI signals before adding any agentic behaviour.

Phase 3 — Intent Classification and Supervisor Dashboard (Months 6–7)

Adding intent classification and the supervisor real-time dashboard. Supervisors could now see, for all active calls, the inferred member intent and current sentiment — enabling targeted coaching during live calls for the first time. Intent data also began populating the operational analytics database.

Phase 4 — Automated Summarisation and CRM Integration (Months 8–9)

The highest-value phase: generative summarisation with direct CRM write-back. This required the most extensive compliance review and UAT — the CRM record is a system of record for regulatory purposes, and automated writes required formal sign-off. Post-deployment, ACW was reduced significantly and agent satisfaction scores improved as the most time-consuming post-call task was largely eliminated.

Section 6

Challenges and Failure Modes

No production deployment is as clean as an architecture diagram. The following challenges are documented not as cautionary tales but as practical reference points for teams undertaking similar work.

6.1 Latency Spikes Under Load

In early load testing, we encountered P99 latency spikes of 4–6 seconds under peak call volume — well above the 2-second target for real-time agent assist. Root cause analysis identified two issues: gRPC connection pool exhaustion during peak load (resolved by pre-warming the connection pool and implementing backpressure signalling), and ASR model cold-start latency when scaling horizontally (resolved by maintaining a minimum warm replica count sized to 70% of peak call volume).

6.2 Domain Vocabulary and Homophones

The base ASR model struggled with pension-specific terminology: contribution acronyms, fund names, and regulatory terms were frequently misrecognised. More problematically, member names and location names produced high error rates that propagated noise into downstream NLP models. The custom fine-tuning pass reduced most of these errors, but a small class of phonetically ambiguous terms (e.g. specific fund names that sound like common words) required post-ASR correction rules.

6.3 Intent Model Confidence Calibration

The initial intent classifier was overconfident on minority classes — rare intents that appeared infrequently in training data were assigned high-confidence scores that were empirically unreliable. We applied temperature scaling during calibration and adjusted the confidence threshold per-intent rather than using a single global threshold. This reduced false-positive intent signals in the agent widget, which had been identified as a key source of agent frustration in the Phase 2 feedback sessions.

6.4 Summary Hallucination

The generative summariser occasionally produced summaries containing action items that were never discussed in the call — a classic hallucination problem. We addressed this through a grounding constraint: the summariser prompt explicitly instructs the model to only include information that appears verbatim in the transcript context, and a post-generation validation step checks summary claims against the transcript using a simpler extractive model. Hallucination rate fell from approximately 8% to under 1% after this change.

Section 7

Results and Measured Outcomes

Outcomes were measured over a 12-month post-deployment period against pre-deployment baselines.

Beyond the headline savings, several secondary outcomes were significant. First-call resolution improved as intent data enabled better routing — calls were directed to agents with the right specialisation more often, reducing transfers. Supervisor effectiveness increased because the real-time dashboard gave them signal across all active calls simultaneously, compared to the previous practice of walking the floor and listening in at random. Operational insight went from anecdotal to quantitative — the organisation now runs weekly intent analysis reports identifying emerging call topics before they become volume spikes.

Agent reception was initially mixed, consistent with broader industry experience of AI assist tool adoption. A structured change management programme — including agent involvement in prompt design and threshold calibration, transparent communication about what the AI could and could not do, and a clear commitment that the AI would not be used for performance evaluation — was important to achieving the adoption levels that made the outcomes possible.

Section 8

Lessons Learned

Deploy streaming infrastructure before NLP. Having the audio pipeline and gRPC transport running in production — processing real calls, generating real transcripts, even if no NLP is attached — gave us invaluable data about call volumes, audio quality variations, and edge cases before we introduced model complexity.

Fine-tuning on domain data is not optional. Base ASR and NLP models trained on general corpora perform significantly worse on specialised financial vocabulary. The fine-tuning investment (approximately 6 weeks of data engineering and model training) paid back within the first month of reduced error rates in downstream NLP.

Agent trust is a system dependency, not a soft consideration. An agent who does not trust the AI signals will disable, ignore, or actively route around them. The deliberate decision to start with passive display (Phase 2) before adding recommendations (Phase 3) was the right call — agents who had two months of experience seeing accurate sentiment signals were far more receptive to acting on them.

Compliance engagement must be concurrent, not sequential. In a regulated environment, bringing compliance and privacy into the architecture design phase — not after a prototype is built — saves months of rework. Several decisions we made early (data residency, PII handling, model registry requirements) would have been significantly more expensive to retrofit.

Hallucination mitigation in agentic write-back is safety-critical. Any AI system that writes to a system of record — CRM, ERP, regulatory reporting — requires explicit grounding constraints and output validation. This is not a quality-of-life concern; it is a risk management requirement.

Section 9

Future Directions

Several capability extensions are under evaluation or in early development.

Predictive routing. Using the intent classifier output combined with member history to route calls to the optimal agent before the call connects — matching not just by skill, but by inferred member need and historical resolution patterns. Industry data suggests predictive routing can improve first-call resolution by 15–20%.

Proactive self-service deflection. Using intent signals in the IVR to offer targeted self-service options at the moment of highest relevance — before the member commits to waiting for a live agent. Member self-service resolution rates have room to improve significantly with better timing.

Multimodal agent assist. Integrating the real-time call context with the RAG-based knowledge retrieval system being built in parallel — surfacing the specific knowledge base article or policy document relevant to the current call topic without the agent needing to search manually.