The thesis — voice-first financial AI that speaks your language
The AI that helps you with your finances should not only listen — it should listen in your language, understand your culture, and take real-world actions for you.
Two users, one capability
A North American user taps a button and says “negotiate my internet bill”. An immigrant professional taps the same button and says “el bill from last month is too high, ayúdame”. Both interactions work equivalently.
That is the thesis. For financial AI to be useful to the people who actually need it, multi-lingual is first-class from day one — not a V2 roadmap item.
Picture a bilingual concierge at a major international hotel. They listen to you in whatever language you arrive in, switch to the staff’s language to handle your request, then answer you back in yours. The switching is effortless. The service is the same regardless of what you speak.
A family of capabilities
Bill-negotiation is the hero demonstration in this course because it is the most concrete — you can hear the agent on a real phone call. But it is one capability inside a larger family.
Call a telecom or insurance provider, navigate their phone tree, and lower a recurring monthly cost.
Walk through a statement line by line in the user’s language — what each charge is, why it changed, what looks unusual.
Have a conversation about spending patterns, suggest cuts, and set up reminders — in plain language, not finance jargon.
When the question gets out of scope, hand off to a real financial advisor with a calendar invite and a one-page summary of context.
Remind about due dates before they hit, and summarise the year’s spending for tax prep — in the format the user’s tax software expects.
Meet the actors in the demo
For the rest of this module, focus on three actors. Subsequent modules expand the cast to the full architecture.
The user. A bilingual professional who code-switches naturally between Spanish and English in the same sentence.
The voice assistant. Talks to Alex in Spanish, talks to the provider rep in English, and bridges the two over SIP telephony.
A human in the provider’s retention department. They speak English. They have discounts to give.
Follow the conversation
Here is a 90-second negotiation as a group chat. Notice the agent switching languages depending on who it is talking to — Spanish with Alex, English with the rep.
How the assistant knows which company to call
To call the right number for the right provider, the assistant looks the answer up in a database table — not a hard-coded list. Here is the actual Python code, exactly as it lives in the codebase.
# Look up provider phone from directory
dir_q = (
db.table("provider_directory")
.select("*")
.eq("provider_name", body.provider_name)
.eq("category", body.category)
.eq("is_active", True)
)
if body.subcategory:
dir_q = dir_q.eq("subcategory", body.subcategory)
# Prefer retention department
dir_result = dir_q.order("department").execute()
entries = dir_result.data or []This is how the assistant looks up which company to call.
Start building a question for the database.
Aim it at the table called provider_directory.
Give us every column of whatever rows match.
Match rows where the provider is what the user asked for.
And the category matches too (e.g. “telecom”, “insurance”).
And skip anything marked inactive.
(closing the filter block)
If the user also picked a subcategory like internet vs. mobile.
Narrow the search further with that too.
Sort so the retention department comes first — they have the most discount-granting power.
Actually run the query against the database.
Grab the matching rows, or an empty list if nothing matched.
The backend has no if provider == "Acme Telecom": phone = ... branch anywhere — every fact about every provider lives in a row of a table. Adding a new provider is a one-line database insert, not a code change. Mature systems push as much logic as possible out of code and into data.
The shape of one provider row
A row in provider_directory carries everything the assistant needs to actually place the call — the number, the menu path, and the right department.
('<Provider>', 'telecom', 'internet', '<E.164 phone>', 'retention', 'CA',
'<verified IVR navigation path>',
'{"weekday": "...", "weekend": "..."}', true),Provider name, category, subcategory, and a retention phone number in E.164 format.
The ivr_path tells the agent how to navigate the IVR — which keys to press and what phrases to say to reach a human in retention.
Business hours stored as JSON, and a flag marking the entry active.
What the rest of this course will teach
Negotiation is the visible tip. Underneath sits an architecture that makes any of the capability family work — explaining a statement, coaching a budget, booking an advisor, or summarising a year for tax prep.
Next up: meet the cast that makes it happen — the seven components, including the language detector that decides which voice the agent uses on each turn.
Meet the cast — seven components that make the assistant work
Each one does one job. Together they make a conversation happen.
An interpretation booth, not an AI
Picture a simultaneous-interpretation booth at a major international summit. Interpreters translate live, clerks log every exchange, archivists hold the master directory of attendees, security keeps the consent paperwork, audio engineers patch the room into the broadcast feed, language analysts identify which language a speaker just switched into, and a final translator turns the spoken record into a written report.
Seven specialists, seven jobs, one outcome. The voice-first financial assistant works the same way — no single component is "the AI."
Saying "the assistant broke, fix it" gets a guess. Saying "the agent joined the room but the extraction LLM never wrote a result row" gets the actual fix. The vocabulary of the cast IS the debugging vocabulary.
The seven cast members
Each card below is one actor, one job, one component type. Colors stay consistent across the whole course — every time you see teal, it's the Directory; every time you see plum, it's the Agent.
Backend
Receives HTTP requests, writes DB rows, orchestrates the call. Doesn't speak. Doesn't listen.
Python / FastAPI service
The Directory
Master table of every provider the assistant can call — retention numbers and menu paths. Shared across all users.
Postgres table
Language Detector
Reads the first few hundred milliseconds of user audio, decides which language models to route to. Streams continuously with a confidence threshold.
streaming detector + fallback rule
The Agent
A separate long-running process. When a voice-* room opens, it joins and runs the conversation: STT → LLM → TTS.
voice worker process
SIP Telephony Bridge
Translates internet audio into real phone-network audio. The plumbing that lets a digital agent dial a real number.
a SIP telephony bridge
Consent Ledger
Every grant is one immutable row: timestamp, user ID, action type, jurisdiction, and the exact language the consent copy was shown in.
Postgres append-only table
Extraction LLM
After the call, reads the transcript and fills a structured JSON template: new price, savings, confirmation number, effective date, outcome.
post-call structured-output LLM
The Backend's front door
The FastAPI router exposes eight endpoints. Reading the docstring tells you everything the Backend can do — no more, no less.
"""
Sessions router — voice-first financial sessions via outbound SIP calls.
Endpoints:
POST /sessions — create a session request
POST /sessions/{id}/consent — grant consent and start the call
POST /sessions/{id}/cancel — cancel a session
GET /sessions — list user's sessions
GET /sessions/stats — user's outcomes summary
GET /sessions/{id} — session detail + result
GET /sessions/{id}/listen — get a token for live listen-in
GET /sessions/{id}/transcript — full call transcript
"""A file-level note saying "this file is the sessions router." Other developers and AI agents read this first.
Reminds the reader that sessions happen via outbound phone calls — real calls, not chat.
The word "Endpoints" signals a list of URLs this file answers to.
Create. The user taps a button; the Backend writes a row and hands back an ID. No call yet.
Consent. This is how the user says "yes, I authorize this call." Only after this does the agent join the room.
Cancel. Back out before (or during) the call. Writes a status, tells the agent to leave.
List. Show me all my past and pending sessions — the history tab of the app.
Stats. One aggregated number across every completed result.
Detail. Everything about one specific session, including the LLM-extracted result.
Listen. A token that lets your phone silently join the audio room — live eavesdropping on the agent.
Transcript. The word-by-word record of who said what. Read-only, pulled from the transcripts table.
End of the docstring. That's the Backend's entire job description.
The Agent's identity file
The agent is a worker — a separate always-on program. When it's handed a session, it constructs a class instance that remembers which session it's working on and what it's supposed to say.
class FinanceVoiceAgent(Agent):
"""Voice agent that handles financial sessions over a real-time room."""
def __init__(self, session_data: dict):
self._session_data = session_data
self._session_id = session_data["id"]
self._call_start = time.time()
super().__init__(
instructions=_build_prompt(session_data),
)Declare the agent's blueprint. The (Agent) part means "inherit from the framework's Agent base class" — we get all of its voice-call machinery for free.
A short description of what this class is for. Future-you will thank past-you.
The setup function that runs once when the agent is created for a specific call. It's handed a dictionary of data about the session.
Remember the whole data packet so the agent can look things up mid-call (provider name, target outcome, user preferences).
Remember the session ID specifically — the agent needs this to write transcripts to the right row in the DB table.
Snapshot the start time. Later we'll use this to compute total call duration.
Call the parent class's setup with one key argument: the instructions.
Build the prompt from the session data — this is the "script" the agent will follow. Different providers and goals produce different prompts.
Close the setup call. The agent is now alive, armed with an ID and a script.
How the worker registers itself
The worker process registers a name with the platform. When a room with a matching prefix opens, the platform dispatches this worker to join it. The agent name is the routing key.
if __name__ == "__main__":
cli.run_app(
WorkerOptions(
entrypoint_fnc=entrypoint,
agent_name="finance-voice-agent",
)
)The standard Python "only run this when the file is executed directly" guard.
Hand control to the framework's CLI runner — it owns the long-running event loop.
Pass in a configuration object describing this worker.
Tell the framework which function to call when a job arrives — the entrypoint sets up the room.
Register under the name finance-voice-agent. Rooms whose dispatch matches this name route to this worker.
Close the options.
Close the runner call. The worker is now alive and waiting for a session.
The aha
This is called separation of concerns. The Backend handles web requests. The Agent handles conversation. The Directory holds shared knowledge. The Language Detector picks the right models. The SIP bridge handles audio. The Consent Ledger records permission. The Extraction LLM turns talk into data. Each one is small enough to understand, replace, or debug on its own — and that is the only way a system this complex stays maintainable.
Consent and compliance — why a phone call has its own legal layer
Running a voice AI that takes real-world actions on behalf of users means serving five different regulatory regimes at once. Here is how the architecture handles it.
A notary signing ceremony that adapts to the country
A notary does not just witness a signature. The proof they create must satisfy whichever jurisdiction will later read the record. In one country the notary needs a witness. In another, a sealed recording. In a third, a written acknowledgement in the signer’s native language.
A voice AI placing a phone call is the same act — a user authorising something real — and the proof requirements shift across borders. That is why consent is not a checkbox. It is a jurisdictional decision made inside a dedicated endpoint.
Two boxes, both must be checked
The first line of the endpoint is the whole philosophy of the feature. Two separate flags. Either one missing, the request dies.
if not body.granted or not body.recording_consent_acknowledged:
raise HTTPException(400, "Both consent and recording acknowledgment are required")body.granted is the box saying “yes, place the call.” recording_consent_acknowledged is a separate box saying “yes, I understand this call may be recorded.”
If either is false, respond with HTTP 400 and refuse to continue. No call. No retry. Done.
Fail-closed means “if anything is unclear, say no.” Fail-open means “if anything is unclear, say yes.” For anything with real-world consequences — money, calls, deletes — pick fail-closed. The ATM does not dispense cash if it is not sure who you are.
Fail-closed on infrastructure
Consent is not enough. The backend also checks that the telephony bridge and the voice platform are actually configured. If either is missing, the call refuses to start.
# Check telephony + voice platform are configured
if not settings.telephony_configured:
raise HTTPException(503, "Outbound calling is not configured. Contact support.")
if not settings.voice_platform_configured:
raise HTTPException(503, "Voice infrastructure is not configured. Contact support.")Check the environment. Are the credentials for the SIP telephony bridge present and valid?
If not, return HTTP 503 with a clear message. Do not guess. Do not silently skip.
Do the exact same check for the voice platform. Two independent dependencies, two independent guards.
Consent is a jurisdictional decision
Before the backend writes anything, it resolves which regulatory regime applies. Country of residence, state overlay for the United States, and the declared processing purpose together pick the consent shape. Here is the decision surface the architecture must encode:
The regulatory summaries below reflect publicly available documentation and are intended as architectural requirements the system must be capable of handling. Specific legal obligations depend on your product, processing activities, data-subject location, and jurisdictions of operation. Verify every claim against current law and qualified counsel before relying on any of it for a production deployment.
Canada — PIPEDA
United States — FCC federal + state overlay
European Union — GDPR + ePrivacy
United Kingdom — UK GDPR + PECR
India — DPDPA 2023
Brazil — LGPDCanada (PIPEDA). One-party recording consent in most provinces, with sector-specific exceptions in BC and Ontario for certain industries. Retention cap of seven years for financial records.
United States. Federal baseline is one-party. But California, Florida, Illinois, Maryland, Massachusetts, Montana, New Hampshire, Pennsylvania, and Washington require all-party consent. The backend must resolve user state to the correct consent mode before dialling.
European Union (GDPR + ePrivacy). Consent must be explicit, specific, informed, and withdrawable at any time. The lawful basis for processing must be recorded alongside the consent.
United Kingdom (UK GDPR + PECR). Mirrors the EU on consent substance. Different supervisory authority: the ICO.
India (DPDPA 2023). Verifiable consent, captured in a language the data-subject understands. The retention purpose must be stated at collection time.
Brazil (LGPD). Purpose limitation is strict: consent is tied to the declared processing purposes. Rectification requests get a 15-day response window.
The consent text shown to a Spanish-speaking user must be in Spanish — not a translated summary, not an English dialog with a Spanish button label. The ledger stores either the exact text or its hash so auditors can later verify the precise words the user saw at the moment of agreement.
The consent ledger — a notary’s record book in SQL
Every granted consent writes one row. The row is tamper-evident: the last column is a hash of everything above it, chained to the previous row. If anyone edits a field after the fact, the chain breaks and the audit surfaces it.
CREATE TABLE consent_ledger (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
user_id UUID NOT NULL,
recorded_at TIMESTAMPTZ NOT NULL DEFAULT now(),
action_type TEXT NOT NULL,
jurisdiction TEXT NOT NULL, -- e.g. "CA-ON", "US-CA", "EU-DE", "IN"
consent_mode TEXT NOT NULL, -- one_party | all_party | explicit_gdpr | verifiable_dpdpa
consent_text_language TEXT NOT NULL, -- BCP-47 tag, e.g. "en-CA", "es-MX", "pt-BR"
consent_text_hash TEXT NOT NULL, -- sha256 of the exact text shown
user_ip_country TEXT NOT NULL, -- ISO-3166-1 alpha-2 at time of consent
granted_granularly JSONB NOT NULL, -- { "place_call": true, "record_call": true, ... }
lawful_basis TEXT, -- required for EU / UK rows
retention_purpose TEXT, -- required for IN / BR rows
previous_entry_hash TEXT, -- chain pointer to prior ledger row
ledger_entry_hash TEXT NOT NULL -- sha256 over all fields + previous_entry_hash
);
CREATE INDEX consent_ledger_user_time ON consent_ledger (user_id, recorded_at DESC);
CREATE INDEX consent_ledger_jurisdiction ON consent_ledger (jurisdiction);Each row is a signed, stamped, language-aware record of exactly one consent event for exactly one user. consent_text_hash means the auditor can prove the precise words shown — not a paraphrase.
granted_granularly stores which specific items were agreed to as a JSON object, so later you can answer “did the user agree to recording?” independently of “did the user agree to placing the call?”
The last two columns form a hash chain. Every new row hashes itself plus the previous row, so any quiet after-the-fact edit breaks the chain and fails audit.
Trace the waterfall
Watch what actually happens when the user taps Start. Six steps, in order. If any one fails, the flow stops cold.
Naming the session and dispatching the agent
Once consent passes and the jurisdiction is resolved, the backend builds a unique session name and asks the voice platform to create it. It also attaches an instruction: dispatch the finance-voice-agent into that session.
room_name = f"fva-{negotiation_id[:8]}-{int(time.time())}"Build a session name that is unique and debuggable: a short prefix, the first 8 characters of the negotiation id, and the current timestamp.
A human reading logs can instantly tell which call this was and when it happened.
await voice.room.create_room(
CreateRoomRequest(
name=room_name,
empty_timeout=600,
agents=[
RoomAgentDispatch(
agent_name="finance-voice-agent",
)
],
)
)Ask the voice platform: create a session with this name.
empty_timeout=600 means “if nobody joins within 600 seconds, tear the session down.” Safety net against leaks.
Attach an agent dispatch: when the session exists, wake up the worker named finance-voice-agent and send it in.
Stamp the ledger, start the dial
Last step: write the hash-chained audit row and kick off the call as a background task so the caller does not sit waiting.
# Write a tamper-evident consent ledger entry
prev = await ledger.last_entry_hash_for_user(user_id)
entry = {
"user_id": user_id,
"action_type": "outbound_voice_call",
"jurisdiction": jurisdiction,
"consent_mode": consent_mode,
"consent_text_language": language_tag,
"consent_text_hash": hashlib.sha256(consent_text.encode()).hexdigest(),
"user_ip_country": ip_country,
"granted_granularly": {"place_call": True, "record_call": True},
"lawful_basis": lawful_basis,
"retention_purpose": retention_purpose,
"previous_entry_hash": prev,
}
entry["ledger_entry_hash"] = hashlib.sha256(
json.dumps(entry, sort_keys=True).encode()
).hexdigest()
await ledger.insert(entry)
# Trigger the dial as a background task
from app.services.voice_service import initiate_sip_call
background_tasks.add_task(initiate_sip_call, negotiation_id, db)Build the ledger entry: who, what, where (jurisdiction), which language, and a hash of the exact consent text shown. Attach the previous row’s hash so the chain is unbroken.
Hash the whole entry and write it. If anyone ever asks “did this user really agree, in this language, under this regime?” there is a signed row answering yes.
Queue initiate_sip_call as a background task. The HTTP response goes out immediately; the actual phone dial happens on its own afterward.
What is next
Consent is granted. The jurisdiction is recorded. The session exists and the worker has been dispatched. Next, we follow the agent into the call — STT, LLM, TTS, and the function tools that let it actually negotiate.
The live call — STT, LLM, TTS, and the plumbing in between
A voice agent is never one model. It is a three-stage pipeline plus a phone bridge, tuned for sub-second round-trip latency.
The interpreter booth at an international summit
Picture the simultaneous interpreter booth at a global summit. A delegate speaks; a headset feeds the interpreter the audio; the interpreter reads, decides, and speaks into a mic; the mic pipes their voice to the other delegates. All in one breath, both directions, for the whole session.
The agent is an AI interpreter booth plugged into a phone line. The trick is that a voice agent is never one model — it is a three-stage pipeline plus a telephony bridge. Let us open the booth.
Inside the agent: three specialists
Zoom in on the agent avatar and you find three specialists working as a team. Each one does one job and hands off. Abstracting them as separate roles matters — each is a swappable vendor, and each must be chosen per language.
STT (speech-to-text) turns the other human's voice into plain text, one phrase at a time. Which STT we use is routed by the Language Detector from Module 2 — no single STT is best across every language.
The LLM (large language model) reads the text, decides what to say back, and writes a reply. It never hears audio — only text. We use a frontier reasoning LLM here because the conversation branches matter.
TTS (text-to-speech) turns that reply into actual audio that goes down the phone line. Voice-matching per language is a real product concern: the “right” voice model for Spanish is different from the right one for Hindi.
Watch the pipeline work, live
Below is a single turn of a call as a group chat. The provider speaks. STT types. LLM replies. TTS speaks. Then the provider asks to verify the account — and the agent pauses to hand the microphone to you. Click Play All.
Assembling the agent in one object
This is where the three specialists get plugged into the same session. The extra settings control when the agent decides the other human is done talking — the difference between a natural conversation and an agent that trampling on sentences.
session = AgentSession(
stt=_get_stt(),
llm=_get_llm(),
tts=_get_tts(),
vad=silero.VAD.load(),
preemptive_generation=True,
turn_handling={
"endpointing": {
"min_delay": 0.6,
"max_delay": 3.0,
},
},
)Build one object called session — the whole voice agent, wired up.
stt is the ear. Hears speech, outputs text. Routed per language.
llm is the brain. Reads text, writes a reply.
tts is the mouth. Turns the reply back into audio.
VAD — voice activity detection. It flags which chunks of audio contain speech vs. silence. silero is an open-source VAD.
Generate the reply early, even before the human finishes — cuts latency.
Endpointing: how the agent decides the other human is done talking.
Wait at least 0.6 seconds of silence before replying.
And never wait more than 3 seconds — if the rep trails off, the agent jumps in.
Short pause = rude interruptions. Long pause = awkward dead air.
The budget splits across stages: endpointing ~600ms, STT finalization ~150ms, LLM first-token ~400ms (streamed), TTS first-chunk ~200ms. These overlap because the pipeline streams — LLM tokens flow into TTS while STT is still finalizing the tail of the utterance — so end-to-end feel is under one second. Blow this budget and the agent feels broken, no matter how good the reasoning.
Picking the ear and the mouth
STT and TTS are separate model families with separate vendors. Swap the TTS and the agent gets a different voice — nothing else changes. The helper functions below exist so the rest of the codebase never hardcodes a model name — per-language routing, cost experiments, and vendor failover all happen here.
def _get_tts():
if _use_hosted_inference():
return inference.TTS(model="<neural voice synthesis model>")
return tts_vendor.TTS()
def _get_stt():
if _use_hosted_inference():
return inference.STT(model="<production STT model>")
return stt_vendor.STT(language="en")A helper that returns the mouth (TTS) for the agent.
If we are routing through a hosted inference gateway...
...use the production neural voice synthesis model — tuned for low-latency, phone-quality speech.
Otherwise call the TTS vendor directly (vendor failover path).
A helper that returns the ear (STT).
Same pattern: prefer hosted inference.
...use the production STT model — fast, accurate on telephony audio.
Fallback is the direct vendor, scoped to a specific language.
What you should take away from this helper is the shape — a single factory function per model type, routed by language, with a direct fallback. The specific STT and TTS models vary by language, latency target, and budget, and good teams re-evaluate them regularly as the field moves. The architecture lets you swap one for a better one without touching anything else.
How the agent actually reaches the phone
The agent lives in an internet audio room. The provider lives on a phone number. Something has to bridge them. That something is a SIP trunk — a rented virtual phone line. We tell the voice platform: “dial this number and drop whoever answers into our room as another participant.”
from voice_platform.api import CreateSIPParticipantRequest, VoicePlatformAPI
api = VoicePlatformAPI(
url=settings.voice_platform_url,
api_key=settings.voice_platform_api_key,
api_secret=settings.voice_platform_api_secret,
)
sip_participant = await api.sip.create_sip_participant(
CreateSIPParticipantRequest(
sip_trunk_id=settings.sip_trunk_id,
sip_call_to=row["provider_phone"],
room_name=room_name,
participant_identity=f"sip-provider-{negotiation_id[:8]}",
participant_name=f"{row['provider_name']} Call",
)
)Pull in the two things we need from the voice-platform library.
Open a connection to the voice platform...
...using the URL, API key, and secret stored in settings.
await — pause here until the dial attempt returns, but do not freeze the server.
Ask the voice platform to create a SIP participant — a phone caller — for us.
Use the SIP trunk ID we rent. This is our outbound phone line.
Dial this number. It is the provider’s line.
Drop the answerer into the same room the agent is already sitting in.
Give them a unique ID so we can find them again.
A human-readable label — helpful in logs and dashboards.
The moment this call returns, the agent and a real human on the provider side are on one audio room.
The pause-and-hand-mic trick
When the provider says “can I verify your name?”, the agent must not invent an answer. It must hand the microphone to you. The way an LLM does that is by calling a function tool — a normal Python function that the LLM knows exists.
@function_tool()
async def request_account_verification() -> str:
"""Call this when the provider representative asks to verify account
information (name, address, account number, PIN, etc.).
This will prompt the user to speak and provide the information.
"""
return (
"The account holder has been prompted to provide verification information. "
"Please wait a moment for them to speak, then continue the conversation."
)Mark this function as callable by the LLM.
A defined function, marked async, that returns text.
This docstring is the instruction manual the LLM reads.
It tells the LLM exactly when this tool is the right move.
The function returns a canned sentence...
...telling the LLM “be quiet and wait, the human is about to talk.”
The agent never invents account info. It hands the mic to you.
Here is the companion tool — the one the LLM calls when the conversation reaches a clean outcome and the call should wrap up:
@function_tool()
async def end_call(
outcome_summary: str,
new_price: float | None = None,
confirmation_number: str | None = None,
) -> str:
"""Call this when the conversation has reached a clear outcome
and the call should end. Pass a short summary, plus the new price
and confirmation number if one was agreed.
"""
await _record_outcome(outcome_summary, new_price, confirmation_number)
return "Outcome recorded. You may now say goodbye and end the call."Same decorator — the LLM can call this one too.
Takes three arguments the LLM must fill in from the conversation...
A one-line summary of how the call went.
The new monthly price, if one was agreed (otherwise None).
The confirmation number the rep gave, if any.
Docstring tells the LLM exactly when to fire this tool.
Write the outcome to the database — this is what Module 6 picks up.
Give the LLM permission to deliver a polite goodbye.
An LLM by itself can only generate text. The moment you want it to do something — look up a record, pause the conversation, save a result, end the call — you give it a function tool. The LLM decides when to call it; your code decides what happens when it is called. That split is what turns a chatbot into an agent that gets real work done.
The agent’s personality lives in one string
Everything the agent believes about itself — its identity, its job, its tone, the deal it wants — is stored in a system prompt. Tweak the string; the agent behaves differently on the very next call. No retraining, no deploy of a new AI.
Exact wording evolves on every production team — what matters for learning is the shape. Every prompt in production follows a sectioned layout like this:
Identity
“You are a professional financial voice agent acting on behalf of a customer.” The LLM adopts this role from line one.
Situation
Current plan, current price, target price, best alternative offer — filled in per call via {placeholders}.
Call protocol
Recording announcement, IVR navigation, opening line, closing checklist — the script structure, not the tuned wording.
Verification policy
When the provider asks for account info, the agent hands the mic to the user. Never guess, never fabricate.
Boundaries
Never agree to a higher price. Never be rude. End politely if the rep is hostile. Escalation ceiling, stated up front.
The five-section layout above is universal — every production voice agent follows something like it. The exact talking-point order, the cultural tone per locale, and the phrasing that actually converts on a call are things teams learn by running thousands of real calls and revising the prompt. How a direct-ask opening lands in one culture but needs a relationship-first preamble in another is not a line you read once; it is a measurement you keep improving.
Next: beyond one language
So far we have touched multi-lingual lightly — a routed STT, a voice-matched TTS. That understates the problem. In Module 5 we go deep: code-switching mid-sentence, cultural adaptation of negotiation tactics, per-locale financial vocabulary, and how we evaluate quality when the agent speaks a language the engineering team does not.
Beyond one language — the depth of multi-lingual voice AI
The easy part is picking a Spanish voice for the agent. The hard part is code-switching, cultural adaptation, and the fact that “dos mil” and “$2000” are the same number spoken in two different ways.
A diplomatic translator, not a bilingual tourist
Picture a diplomatic translator at an international conference. They are not just bilingual — they are bi-cultural, trained on the specific vocabulary of the subject being discussed, and they know when a direct translation would be rude or confusing. A generic translator would fail in the room; a specialist elevates the conversation.
That is what a multi-lingual voice agent must be. Speaking the language is the floor. Serving the user well is the ceiling.
The agent is a domain-specialist interpreter in a glass booth — financial vocabulary on one side, cultural register on the other, and a real-time audio feed in the middle. Everything in this module is about what the booth actually has to do.
Why monolingual voice AI fails a majority of its market
The demographic reality is blunt. A voice agent that can only hear and speak English is built for the smaller half of the relevant user base — especially in financial services, where new arrivals, diaspora families, and multi-generational households dominate the onboarding funnel.
An assistant for financial conversations is exactly where a user wants their first language — numbers, contracts, and consent are cognitively expensive in a second language. The audience that most needs the assistant is the audience that English-only voice AI cannot serve.
The three genuinely hard problems
Picking a Spanish voice model is not hard. These three are.
1. Code-switching
Users mix languages mid-sentence, sometimes mid-word. “Quiero cancelar my subscription porque ya no lo uso.” The STT layer must emit per-token language labels, not a single primary language.
2. Cultural adaptation
A direct “threaten-to-switch-providers” opening works in North America. In many Asian markets, relationship-building first is more effective. The agent must change not just language, but register.
3. Financial vocabulary per locale
RRSP ≠ 401(k) ≠ ISA ≠ PPF ≠ PEA. Even within English: HSA vs FSA. Even within Spanish: the vocabulary differs between Mexico and Spain. The agent must map concepts across locales.
How the Language Detector actually works
When a call opens, the first job is to route the audio to the right models. The Language Detector reads a short probe of the user’s speech, decides on an ISO language code, and hands that code to the rest of the pipeline.
async def route_by_language(audio_stream, user_profile):
"""Probe the first audio segment, then route to per-language models."""
probe = await audio_stream.take_first(ms=800)
detection = await language_detector.detect(probe)
if detection.confidence < 0.75:
iso_code = user_profile.preferred_language
else:
iso_code = detection.iso_code
stt = stt_factory(iso_code)
tts_voice = voice_factory(iso_code)
return PipelineConfig(
stt=stt,
tts_voice=tts_voice,
llm_locale=iso_code,
)Define an async function that picks the right models based on what the user sounds like.
A one-line docstring so future-you remembers what this does.
Take the first 800 milliseconds of audio — long enough to detect a language, short enough not to delay the response.
Ask the detector what language this sounds like. It returns a code and a confidence score.
If the detector is not sure (less than 75% confident)…
…fall back to the language the user set in their profile. Better a known answer than a guess.
Otherwise…
…trust the detector’s answer.
Pick the streaming STT model for that language (e.g. es-MX → Spanish-Mexican STT).
Pick a TTS voice whose accent matches the user’s language. An agent that listens in Spanish but replies in an American English accent feels wrong.
Bundle all three choices into one config object…
…one for how we listen…
…one for how we speak…
…and one for which locale the reasoning LLM should prefer for vocabulary and formatting.
Return the config so the rest of the pipeline can be built on top of it.
Eight hundred milliseconds is the smallest probe window where detection quality stabilises on natural speech; going lower loses accents and intonation cues. A 0.75 confidence threshold is set below where we trust the detector alone and above where we would rather use a known good answer. Both numbers are tuned per deployment, not guessed.
Code-switching — a mixed-language turn, end to end
Walk through what happens when the user mixes languages in one sentence. Each component in the pipeline has to handle the fact that “primary language” is not a single value for this turn.
Some require a single primary language declared up front — they will simply mis-transcribe tokens in the other language. Others stream per-token language labels. The pipeline uses the latter because the first approach fails on the demographic we serve.
Cultural adaptation — four openings for the same negotiation
A call to a retention line starts the same way across cultures: long-tenured customer, looking at options. The opening move, however, is not the same. These are four culturally-tuned opens for the same business goal.
North American — direct
“Hi, I am calling about a long-term customer’s internet pricing. They have seen competitive offers and would like to discuss options.”
leverage: churn threat
Latin American — warm
“Buenos días, le habla de parte de un cliente de muchos años. ¿Cómo está usted hoy?”
leverage: loyalty, relationship
South Asian — respectful
“Sir/ma’am, namaste. We are following up on a customer’s long tenure with your service and would be grateful for a few minutes.”
leverage: respect, long association
East Asian — formal
“I am representing a long-standing customer. We would like to respectfully inquire about retention options available at this time.”
leverage: formality, face-preserving ask
Translating the North American script word-for-word into Spanish produces an agent that sounds pushy or disrespectful in a Latin American context. The agent’s prompt template is culture-aware, not just language-aware — a different opening, a different pacing, a different answer to “¿Cómo está usted?” before any price gets mentioned.
Financial vocabulary per locale
The agent must recognise parallel financial concepts across countries, map them onto the user’s locale, and use the correct local term when speaking. “Retirement account” means a different specific thing in every jurisdiction below.
The agent’s prompt template carries the locale’s vocabulary list as a first-class input. Ask it about “my retirement account” in Canada and it reasons about an RRSP; ask the same thing in the UK and it reasons about a SIPP. Same concept, different word, different rules.
Evaluation and cost
Two things matter once the pipeline works: how do you know it is actually good, and what does each minute cost?
Automated metrics do not catch register mistakes, cultural misses, or subtle mistranslations of financial concepts. QA for this pipeline includes native speakers per supported language, reviewing sampled transcripts against a rubric: correctness, register, cultural appropriateness, vocabulary accuracy.
Spanish and French STT pricing is typically near parity with English. Lower-resource languages like Tagalog, Swahili, and Urdu often cost 1.5–3× more per minute from commercial vendors. This variance feeds directly into per-market unit economics and informs which languages ship first.
A transcript-accuracy score can be 98% and the agent can still sound rude in Japanese. Automated metrics measure words; humans measure trust.
Where we go next
You have now seen what it takes to make the assistant speak, listen, and understand across languages — detection, code-switching, cultural adaptation, per-locale vocabulary, and the evaluation that keeps it honest. In the final module, we follow the multi-lingual conversation all the way to its structured destination: a single row in a database, ready to be shown back to the user.
From conversation to structured financial data
A four-minute call becomes one row in a database — across any language it was spoken in.
A court reporter and a paralegal
You have seen the call happen (Module 4), in English and in many languages (Module 5). Now we need to turn four minutes of conversation — in any language — into a single structured row a spreadsheet can add up.
The pattern is borrowed from the courtroom. During the trial, a court reporter types every word spoken, labels each speaker, stamps the time. They do not summarize. They just capture. After the trial, a paralegal reads that transcript and fills out a one-page form: verdict, damages, next hearing date, reference number. Two jobs, done in sequence, both needed.
Talks on the phone in real time, in the user's language. Its job is conversation, not paperwork.
_write_transcript)
Writes every spoken turn to the database as the call happens, tagged with the language it was spoken in. Captures, does not think.
Reads the whole transcript — even if it is mixed-language — and fills in a JSON form. A different call, a different job.
Stores the clean row in normalized form: ISO currency codes, ISO-8601 dates, constrained vocabularies. Everyone downstream reads from here.
The opening scene
The agent just hung up. The call lasted four minutes and produced about 800 words of transcript. Some of those words were English, some were Spanish, some were numbers the user said out loud — "me pueden ofrecer setenta dólares al mes, por doce meses, con número de confirmación ABC-123..."
Now we need to turn that into a row a spreadsheet can add up. Here is how.
Capture first, think later
Every time the agent or the provider says something, it gets saved. This is unstructured data landing in a row.
def _write_transcript(self, speaker: str, text: str):
"""Write a transcript segment to DB."""
try:
db = _get_db()
elapsed_ms = int((time.time() - self._call_start) * 1000)
db.table("call_transcripts").insert({
"call_id": self._call_id,
"speaker": speaker,
"text": text,
"lang": self._detected_lang,
"timestamp_ms": elapsed_ms,
"is_final": True,
}).execute()
except Exception:
passEvery time someone speaks, call this function with who spoke and what they said.
speaker is a label — "agent", "provider", or "user". lang is the ISO language code the Language Detector saw for this turn. timestamp_ms is how many milliseconds into the call it happened.
Insert one row per turn into the call_transcripts table. is_final means "this is the committed version, not a draft."
try: ... except: pass is defensive code that says "if the database is having a bad moment, do not crash the live call." A missing transcript line is annoying; a dropped phone call is worse.
The fill-in-the-blanks form
After the call ends, we hand the whole transcript to a prompt template and tell the LLM: read this, fill in these blanks, nothing else.
EXTRACTION_PROMPT = """\
Analyze this call transcript between a negotiation agent and a service provider.
Extract the negotiation outcome.
TRANSCRIPT:
{transcript}
CONTEXT:
- Provider: {provider_name}
- Original price: ${current_price}/month
- Target price: ${target_price}/month
- Primary language: {lang}
Normalize amounts to a float and a 3-letter ISO currency code.
Normalize dates to ISO-8601 (YYYY-MM-DD) regardless of input language.
Extract these fields and respond ONLY with valid JSON:
{{
"outcome": "success" | "partial" | "rejected" | "callback" | "unknown",
"outcome_detail": "one-sentence summary",
"new_price": <float or null>,
"currency": "<ISO-4217 code or null>",
"new_plan_name": "<string or null>",
"savings_monthly": <float or null>,
"savings_annual": <float or null>,
"contract_months": <int or null>,
"promo_expires": "<YYYY-MM-DD or null>",
"confirmation_number": "<string or null>",
"effective_date": "<YYYY-MM-DD or null>",
"agent_name": "<provider rep name or null>",
"extraction_confidence": <float 0-1>
}}
"""This is a template. {transcript}, {provider_name}, {lang}, and the prices are placeholders that get filled in with real values before we send it to the LLM.
The trick is the output shape. We give the LLM a JSON template with named fields and tell it: "respond ONLY with valid JSON."
The two normalization lines are the locale bridge. They tell the LLM: it does not matter whether the user said "setenta dólares," "£70," or "₹5,800" — give me back a float and an ISO-4217 currency code. And it does not matter whether the user said "May 1st" or "a partir del uno de mayo" — give me YYYY-MM-DD.
The outcome field uses a small schema of allowed values: success, partial, rejected, callback, unknown. Five words everyone downstream can trust.
Every number is locale-ambiguous
Currency symbols, decimal separators (, vs .), digit grouping (1,000 vs 1.000 vs 1,00,000), and spoken-number forms all vary. A system that assumes one format breaks silently on the first multilingual user.
def normalize_amount(raw: str, lang: str) -> tuple[float, str]:
"""Return (amount, ISO-4217 currency) for a raw spoken or written figure."""
WORDS = {
"es": {"mil": 1000, "dos mil": 2000, "cinco mil": 5000},
"hi": {"lakh": 100_000, "crore": 10_000_000},
"fr": {"mille": 1000},
}
SYMBOL_TO_ISO = {"$": "USD", "£": "GBP", "€": "EUR", "₹": "INR", "¥": "JPY"}
s = raw.strip().lower()
currency = next((iso for sym, iso in SYMBOL_TO_ISO.items() if sym in s), "USD")
# Spoken-number form: "dos mil" -> 2000
for phrase, value in WORDS.get(lang, {}).items():
if phrase in s:
return float(value), currency
# Written form: strip symbols + locale-aware separators
digits = "".join(c for c in s if c.isdigit() or c == ".")
return float(digits or 0), currencyGiven raw text like "dos mil", "£100", or "₹1,00,000" (the Indian lakh grouping), return a normal float and a 3-letter currency code.
The SYMBOL_TO_ISO map turns $, £, €, ₹, ¥ into USD, GBP, EUR, INR, JPY. One currency column downstream — no ambiguous dollar signs.
The WORDS map catches spoken numbers per language. "dos mil" → 2000. "un lakh" → 100,000. "deux mille" → 2000.
The final fallback strips everything non-numeric and parses what is left. In production this is a safety net — the LLM itself does the heavier locale parsing using the prompt's normalization instructions.
A system that assumes one format breaks silently on the first multilingual user. The extraction layer has to do two things at once: convert words and symbols into numbers, and tag each number with the currency the speaker meant — because $100 in one country is not $100 in another.
Dates across locales, one ISO format
The same story plays out for dates. The user may say it several ways; the database stores one way.
# Input phrases the LLM might see in a transcript:
#
# English: "effective May 1st"
# Spanish: "a partir del uno de mayo"
# French: "à compter du 1er mai"
# Hindi: "एक मई से"
#
# All four must produce the same JSON value:
#
# "effective_date": "<YYYY>-05-01"
LOCALE_DATE_HINT = """\
The transcript may express dates in the speaker's language.
Recognize month names across: en, es, fr, hi, de, pt, ar, zh.
If the year is not spoken, assume the next future occurrence.
Always emit ISO-8601 (YYYY-MM-DD).
"""The four phrases above all mean the same thing: a future effective date, the first of May. The user spoke in four different languages; the database needs one answer.
The extraction prompt includes a locale date hint — a short instruction block that tells the LLM which month names to recognize and how to fill in missing years.
Output is always ISO-8601 — YYYY-MM-DD — so a dashboard, a calendar invite, and an SMS reminder all read the same date the same way.
Why at the LLM layer instead of a date-parsing library? Because date-parsing libraries are per-locale and brittle. A frontier reasoning LLM reads all of these natively.
Handling code-switched transcripts
The hardest case: the user switches language mid-sentence. STT models run per-language. The extraction LLM does not care — it reads mixed-language transcripts natively.
# Excerpt from a real code-switched transcript
# (user speaks Spanish; provider rep responds in English):
# [USER] Hola, llamo sobre mi cuenta.
# [PROVIDER] Sure, let me pull that up. Can I have your account number?
# [USER] Sí, es 4812-7733. Quiero ver si puedo bajar la factura.
# [PROVIDER] I can offer you $70 a month for the next 12 months.
# [USER] Perfecto, acepto. Necesito el número de confirmación.
# [PROVIDER] Confirmation number ABC-123456. Effective the first of next month.
# The extraction LLM returns one clean JSON row:
{
"outcome": "success",
"new_price": 70.00,
"currency": "USD",
"contract_months": 12,
"confirmation_number": "ABC-123456",
"effective_date": "<next-month-first>",
"extraction_confidence": 0.94
}Half the lines are Spanish, half are English. The user's account number shows up in Spanish. The final deal terms show up in English. Both matter for the same row.
The extraction LLM reads the whole transcript as one document and fills in the JSON template. Spanish phrases and English phrases feed the same fields — it never has to pick a language.
This is why extraction lives at the LLM layer and not at the STT layer. A per-language parser would need to know which sentence is which language and stitch results together. The LLM just reads.
STT runs per-language. The extraction LLM reads mixed-language transcripts natively. This is why we extract at the LLM layer instead of at the per-language STT layer — it collapses the multi-lingual problem into a single prompt with a single output shape.
Drag the words into the right slots
This is what the Extraction LLM actually does — only it does all five at once, for the whole transcript, in whatever language the transcript is in. Drag each phrase onto its matching JSON field.
new_price — the monthly dollar amount the provider offered
contract_months — how many months the deal locks in for
confirmation_number — the reference string for later proof
agent_name — which rep handled the call
effective_date — when the new price kicks in (YYYY-MM-DD)
Stitch the transcript, fill the template, call the LLM
Before we can run extraction, we stitch all the transcript rows back into one block of text and plug it into the template.
# Build transcript text
transcript_text = "\n".join(
f"[{s['speaker'].upper()}] {s['text']}" for s in segments
)
prompt = EXTRACTION_PROMPT.format(
transcript=transcript_text,
provider_name=row["provider_name"],
current_price=row.get("current_price") or "unknown",
target_price=row.get("target_price") or "unknown",
lang=row.get("primary_lang") or "en",
)
# Call LLM
from app.core.llm import get_llm_for_feature
llm = await get_llm_for_feature(row["user_id"], "call_extraction")
response = await llm.ainvoke(prompt)
raw_text = response.content if hasattr(response, "content") else str(response)Stitch the rows back into one long string. Each line gets a speaker label in brackets so the LLM knows who said what — even if the languages alternate line to line.
Fill the template's placeholders with the real transcript, the provider's name, the price context, and the primary language the Language Detector tagged during the call.
Ask the LLM system for the model configured for call_extraction, then send the filled prompt. raw_text is whatever the LLM wrote back — usually JSON, sometimes with extra junk wrapped around it.
Strip the wrapper, parse the JSON
Even when you tell an LLM "no markdown," it sometimes wraps its answer in triple-backticks anyway. Before we can parse the JSON, we strip those fences if they are there.
# Parse JSON from response
# Strip markdown code fences if present
clean = raw_text.strip()
if clean.startswith("```"):
clean = clean.split("\n", 1)[-1]
if clean.endswith("```"):
clean = clean.rsplit("```", 1)[0]
clean = clean.strip()
extracted = json.loads(clean)Trim whitespace from the LLM's reply to get a clean starting point.
If the answer starts with triple-backticks (markdown fences), drop the first line — that is usually ```json or just ```.
If it ends with triple-backticks, chop off the closing fence too. Trim again.
json.loads(clean) turns the cleaned text into a Python dictionary we can read like extracted["new_price"]. If the text is not valid JSON, this line crashes — which is exactly what we want, because garbage in a database row is worse than an error.
Messy input (transcript, email, receipt photo) → LLM with a JSON template → clean row in a database → everyone downstream reads the clean row. Once you know this shape, you can ask an AI coding agent for it directly: "Give me an extraction pipeline: input is X, output schema has these fields, normalize to these standards." You get predictable code instead of hand-waving.
Trust, but verify the math
LLMs sometimes forget to do the arithmetic. If the model filled in new_price but left savings_monthly blank, the code computes it.
# Compute savings if not provided
if extracted.get("new_price") and row.get("current_price"):
new_p = float(extracted["new_price"])
cur_p = float(row["current_price"])
if not extracted.get("savings_monthly"):
extracted["savings_monthly"] = round(cur_p - new_p, 2)
if not extracted.get("savings_annual"):
extracted["savings_annual"] = round((cur_p - new_p) * 12, 2)If the LLM filled in a new price and we know the old price, do the subtraction ourselves.
Backfill savings_monthly as current − new, and savings_annual as (current − new) × 12.
Why? Because downstream dashboards will sum the savings. Numbers the LLM skipped must not quietly become zero.
Tell the user what happened
Once the row is safe in the database, we send one push notification that matches the outcome. The message is localized to the user's preferred language — the same one the call was conducted in.
# Send push notification with result
outcome = extracted.get("outcome", "unknown")
savings = extracted.get("savings_monthly")
if outcome in ("success", "partial") and savings:
msg = f"Your negotiation with {row['provider_name']} saved {fmt_money(savings)}/month!"
elif outcome == "rejected":
msg = f"Your negotiation with {row['provider_name']} didn't get a lower rate this time."
elif outcome == "callback":
msg = f"The rep at {row['provider_name']} will call back — we'll follow up."
else:
msg = f"Your negotiation with {row['provider_name']} has been processed."
await _notify_user(call_id, db, "Negotiation Result", msg, lang=row["primary_lang"])Look at the outcome field. Pick a message that matches. Four branches, four possible vibes.
The f-string templates use {row['provider_name']} as a placeholder — at runtime, Provider (or any other name) gets slotted in. No provider name hard-coded.
Because outcome is one of a fixed set of values, this if/elif/else is safe — there is no sixth possibility hiding in the data.
lang=row["primary_lang"] tells the notification layer to render the message in the user's language before delivering it.
The database enforces the vocabulary
The JSON template asks for one of five outcome values. The database enforces the same list — so even if an LLM decides to get creative, the row gets rejected at the door.
CHECK (outcome IN ('success', 'partial', 'rejected', 'callback', 'unknown'))This is a CHECK constraint — a rule on the table itself. Any insert where outcome is not one of these five strings is rejected by the database before it even lands.
Belt and suspenders. The prompt asks nicely; the schema enforces.
The same pattern protects the currency column — a CHECK restricting it to valid ISO-4217 codes ensures no locale mix-up silently corrupts the ledger.
You have seen a real voice-first financial AI, end to end
Across six modules you have seen a voice-first financial AI end-to-end: the thesis, the architecture (including a dedicated Language Detector), a multi-region consent waterfall, a live bilingual call, the depth of true multi-lingual design, and the extraction pipeline that turns four minutes of conversation into a single structured row — in any language the user spoke.
Bill negotiation is the hero demo. The same architecture shape powers statement-explainers, budget coaches, biller concierges, and advisor-booking flows. The pieces — a real-time audio platform, a SIP telephony bridge, production multi-lingual STT, neural TTS, a frontier reasoning LLM, a normalized database — are reusable. This is the pattern Kardoxa Labs builds, for consumer products and for SMB call centres. You now have the vocabulary to describe, debug, and direct any of it.
Let's talk
If you're building something in this space — consumer or SMB — I'd love to hear about it.
Email smohan@kardoxa.com or connect on LinkedIn.
Kardoxa Labs occasionally publishes new architectural breakdowns like this one. To know when the next one drops, follow me on LinkedIn.