Clio Voice AI

What this is

Clio Voice AI is an outbound and inbound sales agent that talks to leads over the phone — or a browser microphone, in Sandbox mode — driven by a conversation agent built on LangGraph and a real-time audio pipeline built on Pipecat. It answers calls, qualifies leads, books meetings, transfers to a human when a conversation needs one, and dials out on campaigns, all while every provider it depends on (the LLM, the speech-to-text engine, the voice, the phone carrier) stays swappable from inside the app itself.

There is no environment-variable credential file to hunt through and no Django admin panel to learn. Every external account this platform talks to — OpenAI or Bedrock for reasoning, Deepgram for transcription, ElevenLabs for voice, Plivo/LiveKit for telephony, Razorpay for billing, Google for sign-in, calendar and mail — is entered once, on a settings page built for exactly that provider, and stored encrypted in the database.

Who does what here

Architecture

One call is one independent operating-system process. There is no shared long-running worker handling every call — when a call starts, whether from the dashboard's Sandbox, a test dial, a campaign, or an inbound webhook, the dashboard spawns a fresh manage.py run_call subprocess for it and walks away. That process builds the full audio pipeline, runs the conversation to completion, writes every cost line and transcript line back to the database as it goes, and exits.

The live call pipeline

Audio flows through a Pipecat pipeline in one direction, with the conversation agent sitting in the middle in place of a typical LLM node:

The agent step does not use Pipecat's own LLM abstraction — it calls a LangGraph graph directly, one node execution per user utterance. Two ways that step can run, chosen by a single app setting:

• Non-streaming — one structured LLM call returns the reply text, the next conversation state, an emotion tag, and any extracted lead details together.
• Streaming — a plain-text call streams the reply to the voice sentence by sentence for lower time-to-first-audio, then a second, smaller call classifies the turn once the reply is already known. Costs roughly 1.9× the tokens in exchange for a faster-feeling conversation.

flowchart LR
    A[Dashboard
Sandbox / Test dial / Campaign / Inbound] -->|spawns| B(("run_call
subprocess"))
    B --> C[Pipecat pipeline]
    C --> D{{LangGraph agent}}
    D --> E[(Database)]
    C -->|per turn| E
    style B fill:#e8a857,stroke:#b8791f,color:#241a08
      

Turning a state machine into a conversation

A script's conversation states become one node each in a LangGraph graph — there's no multi-hop traversal per turn; the graph always enters at whichever state the conversation is currently in, runs exactly that one node, and stops. Each turn's LLM call returns the reply, the next state, a detected emotion and any extracted details together, and the proposed next state is always checked in code against that state's own declared transitions before it's trusted — an invalid or hallucinated value simply keeps the conversation where it is instead of jumping somewhere undefined.

Real actions get a real safety gate

Booking a calendar slot or sending an email is never taken on the model's word alone. Both require an exact match, checked in code, against a slot or address that was already looked up and read back to the caller earlier in the same turn sequence — the model cannot skip straight to "done" without that prior step having actually run. Calls using calendar, email, or message-taking tools run on the slower, non-streaming turn path deliberately: describing a real action's outcome honestly requires knowing the real outcome first, which a low-latency streamed reply can't wait for mid-sentence.

Handing a call to a person

A transfer doesn't just tell the caller "connecting you now" — it dials the business's configured contact as a real second participant into the same call, waits for a real answer, and only then has the AI leave, so the caller and the person stay on the line together afterward. If nobody picks up, the AI is told the attempt failed and carries on with the caller rather than the call simply ending.

An alternative pipeline: speech-to-speech

Some providers — OpenAI Realtime and Google Gemini Live — combine listening, thinking and speaking into one continuous connection instead of the separate STT/LLM/TTS stages above. A call can use either pipeline; the conversation design (states, transitions, guardrails) is the same underneath either way, so a script doesn't need to be rebuilt to move between them.

Two provider systems, not one

Don't conflate these — they solve different problems and live in different apps:

A call can override the platform's active provider for a one-off test (Sandbox and Telephony both offer a picker) without ever touching which provider is globally active for everyone else.

Local Setup

Everything below assumes a checkout with its own virtual environment at env/, working directory at the repo root.

python3 -m venv env
env/bin/python -m pip install --upgrade pip
env/bin/python -m pip install -r requirements.txt

cp .env.example .env
# fill in SECRET_KEY at minimum -- generate one with:
env/bin/python -c "import secrets; print(secrets.token_urlsafe(50))"

env/bin/python manage.py migrate
env/bin/python manage.py sync_app_settings
env/bin/python manage.py createsuperuser
env/bin/python manage.py runserver

Seeding a starter script

A brand-new tenant has no conversation script yet. Two ready-made ones ship as management commands:

env/bin/python manage.py seed_default_script
env/bin/python manage.py seed_debt_advisor_script

First Run & Superuser

Sign in with the superuser account you just created and you land in the regular dashboard as the Owner of a brand-new, empty tenant — the same starting point every real signup gets. From here, the order that actually gets a test call working is:

1. Provider Settings — add and activate at minimum one LLM, one STT and one TTS row, plus a Transport provider if you intend to place a real phone call rather than test in-browser.
2. Scripts — seed or write a conversation script (states, FAQ, objections).
3. Sandbox — place a browser-microphone test call against that script before ever touching a real phone number.

A superuser account is also how you reach the separate superadmin console at /console/ — that's where the one-time platform-wide credentials (Razorpay, Google OAuth) and the plan catalog get configured, independent of any one tenant.

Provider Settings

Every vendor in the call pipeline is a row in one of five tables — LLM, STT, TTS, STS (realtime speech-to-speech), and Transport (telephony/SIP). Adding a provider is filling in a form, not editing code.

Many active rows, not just one

LLM, STT and TTS deliberately allow many rows to be active at once — which ones a given business can actually pick from is controlled separately, by a per-business grant, not by a row's own active flag. This is what lets one business run on a cheaper STT/TTS pairing while another runs on a premium one, side by side, without either seeing the other's options. Transport is the one exception: exactly one LiveKit row is active platform-wide at a time, since it's a single shared telephony credential set, not something a business picks between.

Automatic fallback between providers

Any LLM, STT or TTS row can name another row as its fallback, chained up to five deep and checked for loops at save time. If the active provider fails mid-call — an outage, a timeout, a rate limit — the pipeline automatically moves to the next provider in the chain rather than the call dropping. For a language model, the failed request is simply retried on the backup with nothing lost; for speech-to-text or text-to-speech, the one utterance in flight at the exact moment of failure is skipped and the next one uses the backup. Every switch is recorded on the call's own detail page. A business on a plan that allows it can configure their own fallback chain from their granted providers instead of riding the platform-wide default.

Adding a new LLM vendor without writing code

Any vendor whose API speaks OpenAI's own chat protocol — Groq, Together, Fireworks, DeepSeek, Mistral, OpenRouter, or a self-hosted model server — can be added as a plain provider row, no code change required. Pick OpenAI-compatible (custom endpoint) as the provider type, then fill in the vendor's base URL, API key, model id and pricing. A genuinely new API shape (a new coded speech-to-text/speech-to-speech vendor) still needs a small amount of code, not just a row.

Testing a specific pairing

Sandbox and the Telephony test-dial screen both let you pin a call to a specific provider row rather than whatever's globally active — useful for an A/B cost or quality comparison without disturbing production traffic. Picking a provider for one test call never flips that row's own active flag.

App Settings

Everything about pipeline behavior rather than pipeline identity — turn-taking delays, silence and call-ending thresholds, feature flags, the reporting currency — lives here as one row per tunable constant, grouped into categories like Billing, Conversation Behavior, Turn-Taking & Timing, and Silence & Call Ending.

Changes apply from the next call onward, never mid-call — each call is its own subprocess and reads every setting once, at startup.

Cost Tracking

Every LLM token, transcription second, voice character, and telephony minute becomes its own cost line item the moment it happens during a live call, not summarized after the fact. A call's total is computed live from those line items rather than trusted from a cached field — the cached total only becomes reliable a few seconds after a call finishes, once the pipeline has fully torn down.

Multi-currency by design

Different vendors genuinely bill in different currencies — a telephony carrier in INR, an LLM vendor in USD. Each provider row carries its own billing currency, and every cost line is converted into one configured reporting currency using a cached USD→INR rate that refreshes automatically in the background and never blocks a live call waiting on a network request. Both the original vendor-currency cost and the converted reporting-currency cost are kept, so a real vendor invoice can always be reconciled against what this platform recorded.

Language Support

Twelve languages are modeled platform-wide: English, Hindi, Bengali, Gujarati, Kannada, Malayalam, Marathi, Tamil, Telugu, Punjabi, Odia and Assamese. A script is refused on save unless the account has at least one speech-to-text provider and one text-to-speech provider that both support its chosen language — enforced in code, not just documented.

Support is deliberately conservative: a language is only listed for a vendor once it's actually been confirmed against that vendor's own documentation or a real call, never assumed just because a vendor's model might technically stretch further.

Writing a script in its own script, not transliterated

Text written in Latin letters is pronounced with an English accent regardless of which language it's meant to be — a Hindi greeting should be written in Devanagari, not Romanized. The conversation agent actively reinforces this on every single turn for Hindi calls specifically, since a rule stated once at the top of a prompt was found to not be enough on its own to stop later drift.

Scripts & Conversation Design

A script is one agent's complete definition: a system prompt (its standing instructions, sent on every turn), a greeting (with an optional separate one for inbound calls), a fallback line for when something goes wrong mid-turn, a language and voice, an initial state, and the state machine itself.

States turn a conversation into steps — in each state the agent gets instructions for that moment only, plus a list of which states it may move to next based on what the caller just said. A state ticked as ending the call hangs up once the agent finishes speaking there. A visual editor manages this as a flow; if it can't parse existing data it falls back to a plain JSON view rather than losing anything.

A FAQ and an objection-handling list give the agent facts to answer from and situations to recognize and respond to, matched by meaning rather than exact wording. Entries can be typed by hand, or generated from a source: paste a web page address, upload a document (PDF, Word, text, Markdown, CSV, HTML), or paste text directly, and an AI proposes question-and-answer pairs for a human to review and select before anything is saved — nothing from an import is ever stored automatically.

Business hours, when set, give inbound callers outside them an automatic "we're closed" greeting and keep the agent brief; the same hours also gate what appointment times the agent is willing to book. Caller memory, on by default, gives a returning caller a short summary of their own recent calls with that business so the conversation can pick up where it left off.

Calendar & Booking

With Google Calendar connected, the agent checks real availability and books, reschedules or cancels appointments while the caller is still on the line. Every permission box on Google's consent screen has to be ticked for the connection to actually work — a partial grant looks connected but can't read or write the calendar.

No Google account connected is a fully supported state, not a degraded one: the agent books against Clio's own internal calendar record instead, with identical behaviour and the same guarantees. A booking is only ever confirmed to a caller after the booking step has genuinely succeeded, and every booking gets a short, speakable reference number the caller can use later to reschedule or cancel. Rescheduling and cancelling can optionally require the caller's number to match the one the booking was made with, to stop a guessed reference number from being used to tamper with someone else's appointment.

Email

A business connects either Gmail (send-only OAuth — Clio can never read an inbox) or their own SMTP server, so the agent can email a caller who asks for details, and the team can send manually from the dashboard. Reusable templates with fill-in placeholders keep what goes out consistent. Before sending anything, the agent always reads the address back and waits for a clear yes — an address it isn't confident about gets asked for again rather than guessed at.

WhatsApp

After a booking is made, changed or cancelled, the caller can automatically receive a WhatsApp confirmation from the business's own WhatsApp Business number — a business brings their own Meta account and is billed by Meta directly, at no extra charge from Clio. Messages only ever go out using an approved Meta template, only after the underlying booking action genuinely succeeded, and sending never slows the call down since it happens in the background. "Sent" means Meta accepted the message; delivery itself isn't confirmed.

Campaigns & Leads

A campaign combines a script, a list of leads, and a dialing schedule — a time window and timezone, a maximum number of attempts per lead, a retry interval, and how many calls may run at once. Leads can be added one at a time or imported from a CSV with name/phone/email columns (phone required). A lead that doesn't answer is retried up to the configured limit; a caller who says "call me back later" stays in the queue for a later attempt; a caller who asks not to be contacted again is marked Do Not Call and their number is blocked from every future call, campaign or otherwise. Outbound campaign calling requires a one-time business verification (a registration and tax certificate check) before it can start; inbound numbers need none of that.

Every lead also gets a temperature — hot, warm or cold — computed from a deterministic, configurable scoring rule after each call, so a list can be triaged by who's actually worth following up first.

Transfer to a Human

When a caller asks for a person, or says something a business has marked as needing one, the agent can pass the live call to a real phone number — a genuine telephony-level handoff (see Architecture), not a conversational "connecting you now" that goes nowhere. A business maintains a roster of contacts with a priority order, availability toggle and optional working hours; transfer routing can be scoped by category (a "billing" phrase list routes to a billing contact, for instance) with a general fallback contact for anything else. If nobody in the roster answers, the agent is told the attempt failed and carries on with the caller — apologizing and offering to take a message, for example — rather than the call simply ending.

Analytics

Two dashboards, both filterable by time window, script and campaign. Call Analytics covers answer and completion rates, a funnel showing how far calls get through a script's own states, a script-to-script comparison, reply-latency percentiles, best time to call by hour and day of week, and a breakdown of why calls failed. Cost Analytics covers total spend and cost per minute, broken down by language model, speech/voice, telephony and platform cost, spend over time, the most expensive individual calls, and spend by script or by campaign.

Learn From Calls

Every finished call is checked for questions the agent couldn't answer well from its own script. Once the same kind of question has come up more than once, it becomes a reviewable suggestion — grouped with other differently-worded ways of asking the same thing. Nothing is ever added to a script automatically: a human always writes or approves the actual answer, and nothing a caller said is ever copied in as fact. The same page tracks whether an approved fix actually worked in later calls. Flagging a specific wrong or outdated answer directly on a call's transcript feeds the same review queue. Only a short, scrubbed, generalized version of a question is ever stored — never a transcript excerpt — and it's purged automatically after a configured retention period.

Roles & Team

Every tenant has exactly one true owner — whoever the data already belongs to. Ownership isn't a role you're assigned; it's simply the default for any account with no membership elsewhere. An Owner can invite other people into their tenant with a specific, lesser role, and only the Owner can promote or demote another Admin.

Ranks stack — every role above also has everything the roles below it can do. Inviting someone requires they've already signed up (by any method, including Google); there's no separate invite-email flow.

Wallet & Plans

Every tenant has one Wallet with a single balance, fed from two independent sources that expire differently: a monthly subscription grant (expires at the end of its billing cycle) and one-off top-ups (never expire). Spending draws from whichever bucket is about to expire first.

Switching plans and card mandates

Upgrading or downgrading updates the same Razorpay subscription in place. One real constraint worth knowing before it surprises a customer: a domestic Indian card mandate cannot have its plan changed at all — not immediately, not at the next cycle. That's a Razorpay/RBI rule, not a bug in this platform; only UPI Autopay mandates support an in-place plan change. A domestic-card subscriber who wants a different plan has to cancel and re-subscribe.

Razorpay Setup

Configured once, platform-wide, from App Settings → Billing in the superadmin console — not per tenant. Three values, then a webhook, then a one-click sync per plan.

1. Enter razorpay_key_id, razorpay_key_secret and turn on razorpay_enabled.
2. Register a webhook in the Razorpay Dashboard pointed at your domain, subscribed to at least payment.captured, payment.failed, and every subscription.* event.
3. Paste the webhook's own secret into razorpay_webhook_secret — a separate value from the API key/secret pair.
4. On the Plan Catalog page, click Sync to Razorpay on each plan — this creates the matching Razorpay-side plan and stores its id.

https://your-domain.example.com/webhooks/razorpay/

Google Sign-In

"Continue with Google" on the login and signup pages, configured from Settings → Google in the superadmin console. This is its own OAuth Client, entirely separate from the Calendar/Gmail one below — keep that distinction in mind, since they're easy to conflate and are covered on the same settings page precisely because they're both "the Google credentials," not because they're the same thing underneath.

A user signing in with Google never sees Google Cloud Console, never has a Client ID of their own — the platform's one registered app identity is what every sign-in flows through, and each person's own Google account is what determines whose account it is on this end, exactly like signing into any other site with Google.

https://your-domain.example.com/accounts/google/login/callback/

Calendar & Gmail

A completely separate integration from Sign-In, configured from App Settings → Integrations instead — it lets the AI agent book real meetings on a business's calendar and send real email as that business, on the caller's request. Can reuse the exact same Google Cloud OAuth Client as Sign-In, or a different one; either way it needs its own two redirect URIs registered.

Connecting either is a per-business action, not a platform-wide one — an Admin or Owner clicks Connect on their own dashboard, signs into their own Google account when prompted, and grants access there. The refresh token that results is stored against that one tenant only.

Notifications

Each user controls their own notification categories independently from their own Notification Settings page — muting Business Metric Alerts, for instance, without losing anything else. Preferences are per-user, not per-tenant, so two people on the same team can have entirely different notification settings.

Browser push notifications are a separate, opt-in layer on top of in-app notifications, enabled per device from the notification bell.

Superadmin Console

A separate console at /console/, gated on is_superuser rather than any tenant role — this is platform operation, not any one business's dashboard. It has its own navigation, its own visual shell, and does not share templates with the regular dashboard except where a settings page (Plan Catalog, Provider Settings, App Settings) deliberately reuses dashboard-style cards and tables.

Only a real Django superuser can reach any of this — created with manage.py createsuperuser, never through self-service signup, on purpose.

Bare-Metal Server

The production path this platform is actually built for: three systemd services on a single Linux box, gunicorn bound straight to port 80, no reverse proxy in front by default.

Binding directly to port 80 as a non-root user works through one narrow systemd grant, not setcap on the binary (which would get silently wiped the next time a dependency update rewrites it):

AmbientCapabilities=CAP_NET_BIND_SERVICE
CapabilityBoundingSet=CAP_NET_BIND_SERVICE

Redeploying

cd /path/to/Voice-Agent && git pull
env/bin/python -m pip install -r requirements.txt
sudo systemctl restart bizzbuzz-web bizzbuzz-worker bizzbuzz-beat

Nothing else is needed before the restart — migrations, static files and the app-settings registry are all handled by the service's own pre-start hook.

Backups

A safe, hot SQLite backup command ships with the project — it uses SQLite's own online-backup API rather than a plain file copy, which matters specifically because a bare cp can capture a torn snapshot mid-write. Not scheduled automatically; wire it into cron:

0 3 * * * cd /path/to/Voice-Agent && env/bin/python manage.py backup_db

CI/CD Pipeline

Three GitHub Actions workflows, each with one job: prove the code works, get it onto the server, and skip the manual click on a PR when it's ready.

Why a self-hosted runner

The deploy workflow runs on a GitHub Actions runner installed directly on the production server itself, not one of GitHub's own cloud runners. A server behind office NAT with no port forwarded for SSH can't be reached by an outside runner at all — a self-hosted runner sidesteps that entirely by polling GitHub outbound, so nothing needs to be exposed to the internet to make this work.

A merge doesn't always mean a push event

The health check has to be real

Confirming a service is "active" only proves the process started — it says nothing about whether it can serve a real request. The deploy pipeline's last step instead retries an actual HTTP call to the readiness endpoint, with a short per-attempt timeout so one slow attempt can't silently eat the whole retry budget, and a spoofed Host header so the request passes the same host validation a real browser request would need to pass.

Database

SQLite by default, in WAL mode — a deliberate choice for a single-server deployment, not an oversight. WAL lets a reader and a writer proceed concurrently instead of the whole database file locking on every write, which matters here specifically because the web process and every live call's own subprocess hit the same file at once.

When to consider MySQL

SQLite's real ceiling is horizontal scaling and built-in replication, not raw throughput at this scale. MySQL support is opt-in via one environment variable, meant for a deployment that has genuinely outgrown a single server — not a default recommendation.

DB_ENGINE=mysql
DB_NAME=bizzbuzz
DB_USER=bizzbuzz
DB_PASSWORD=
DB_HOST=127.0.0.1
DB_PORT=3306

Create the database with utf8mb4 at creation time, not as an afterthought — MySQL's plain utf8 is a legacy encoding that silently can't store full-width Unicode:

CREATE DATABASE bizzbuzz CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

What Clio Can & Can't Do

An honest scope statement, kept current as the platform grows — not a sales pitch.

What's built and working today

Not built yet

• No public developer API, SDK or outbound webhooks. Every endpoint today is for the dashboard itself; nothing external can call in or be called out to on an event like a finished call or a new booking.
• No bring-your-own-key model. Provider credentials are configured platform-side, with per-business access grants onto existing rows — not a per-business API key of their own.
• No third-party CRM sync. Leads live inside Clio's own record; there's no Salesforce/HubSpot-style two-way sync yet.
• No AI-to-AI handoff mid-call. A live call can transfer to a human; it cannot hand off to a different, separately-specialized AI agent partway through.
• No touch-tone (DTMF) support. The agent can't send or receive keypad input, and can't navigate a phone menu on a transfer.
• FAQ retrieval is keyword-based, not semantic search, and knowledge import is a one-time review-and-approve flow rather than continuous re-crawling of a live document or site.
• Business hours are weekly, not calendar-aware. A one-off holiday closure has no dedicated setting yet, and a working window can't cross midnight.
• No SMS channel. WhatsApp confirmations exist; plain SMS does not.
• No self-service data export or deletion for transcripts and recordings beyond what an operator can already do directly.

Troubleshooting

Every page redirects to a dead HTTPS port, only health checks work

Caused by SECURE_SSL_REDIRECT forcing an HTTPS redirect on a server that has no TLS in front of it yet — gunicorn bound directly to port 80, no reverse proxy configured. Health-check endpoints are commonly exempted from this redirect, which is exactly why they keep working while every real page 301s to a port nothing is listening on. Set SECURE_SSL_REDIRECT=false until real TLS exists in front of the app.

A reverse proxy sends the wrong OAuth scheme

If Google (or any OAuth provider) rejects a redirect_uri as a mismatch, check what scheme your own server is actually building into that URL — a reverse proxy that terminates TLS but forgets to forward X-Forwarded-Proto: https to the app behind it will make every generated URL come out as http:// even though the outside world sees https://. Confirm the proxy sends that header, or force the scheme independently of what the proxy reports.

Moving credentials between environments

Provider API keys are encrypted with a key derived from SECRET_KEY — and that key is deliberately different between environments. Copying the raw encrypted database rows from one environment to another does not work: the destination will try to decrypt them with its own, different key and fail silently, leaving the provider looking configured but functionally empty. The correct path is to decrypt on the source, transfer the plaintext over an already-secure channel, re-encrypt on the destination with its own key, then delete every trace of the intermediate plaintext file on both ends.

A health check that "still isn't ready" after a full minute

Two separate causes produce this same symptom. First: a request that lands on a worker process still mid-import can hang indefinitely without a per-attempt timeout on the client side, silently consuming the entire retry budget on one attempt instead of the many quick retries intended. Second, and easy to miss because it returns instantly rather than hanging: the request's Host header doesn't match anything in ALLOWED_HOSTS, and the framework rejects it immediately — a health check hitting a bare IP or 127.0.0.1 needs an explicit Host header matching a domain that's actually allowed.

Dark mode text is unreadable on part of a page

If a page mixes a design-system framework with hand-written theme tokens, the framework's own dark-mode variables need to be switched on independently — a custom light/dark toggle doesn't automatically flip a separate framework's palette unless something explicitly tells it to.

Command Reference