Voice & Phone Calls

Octipus includes voice capabilities at two levels: local voice I/O for hands-free interaction, and phone calls for making and receiving actual calls through telephony providers.

Local Voice I/O

Speech-to-Text (STT)

Engine	Type	Description
Whisper.cpp	Local	Local transcription after installing the binary and model
Faster-Whisper	Local	Python + CTranslate2 — optimized for speed
OpenAI Whisper	Cloud	Batch transcription with configured API credentials

API: POST /api/voice/transcribe — send base64-encoded audio, receive transcribed text.

Text-to-Speech (TTS)

Engine	Type	Description
Piper	Local	Local speech synthesis with installed voice files
Kokoro	Local	Local speech synthesis with installed model/runtime
Mistral	Cloud	Configured speech synthesis API
OpenAI	Cloud	Configured speech synthesis API

Wake Word Detection

Engine	Type	Description
Sherpa-ONNX	Local	ONNX-based keyword spotting — fast, private
Picovoice/Porcupine	Local	Local keyword engine requiring an access key
VAD Fallback	Local	Voice Activity Detection — triggers on any speech

Note

All voice features are optional. Octipus works fully without any voice configuration.

---

Phone Calls

Make and receive actual phone calls through telephony providers. Conversations are powered by the same LLM models you use for chat — with a fast, direct path optimized for voice latency.

Supported Providers

Provider	Protocol	What You Need
Twilio	Programmable Voice + Media Streams	Account SID + Auth Token
Telnyx	Call Control v2	API Key + Connection ID
Plivo	Voice API + XML	Auth ID + Auth Token

All three providers support outbound calls, inbound calls, and webhook-based conversation flow. Phone-number configuration and discovery depend on the provider.

Quick Setup

1. Store credentials in the vault

Add your provider’s API credentials in Settings > Vault:

For Twilio: twilio_account_sid and twilio_auth_token For Telnyx: telnyx_api_key and telnyx_connection_id For Plivo: plivo_auth_id and plivo_auth_token

2. Configure the provider

Set two settings (via Settings page or API):

Setting	Value	Example
voice.telephonyProvider	Your provider name	twilio
voice.publicUrl	Your public webhook URL	https://abc123.ngrok.io

3. Assign a model to the “voice” topic

In the Models page, assign a fast model to the voice topic. Measure latency with your model, hardware, and network.

Verify credentials, the caller number, public webhook reachability, and a test call. The expert prompt is loaded from the Communicator expert automatically.

Twilio Setup Guide

1. Create a Twilio account at twilio.com
2. Get credentials from Twilio Console → Account → API keys & tokens:
   • Account SID: starts with AC + 32 hex chars (34 total)
   • Auth Token: 32 hex chars
3. Store in vault: Add twilio_account_sid and twilio_auth_token in Settings → Vault
4. Phone number: Auto-detected from your Twilio account — no manual config needed. Buy a number in the Twilio Console if you don’t have one.
5. Set webhook URL: Must be publicly accessible (use ngrok or Cloudflare Tunnel)
6. Configure Twilio webhook: Point your Twilio phone number’s webhook to https://your-url/api/voice/webhook/twilio

Common Issues

Problem	Cause	Fix
HTTP 401	Auth Token mismatch	Regenerate in Twilio Console → API keys & tokens
HTTP 403	Account suspended, old token, or sub-account mismatch	Verify in Twilio Console
HTTP 404	Invalid Account SID	Check vault secret twilio_account_sid
No phone number detected	Account has no numbers	Buy one in Twilio Console
Webhook timeout	Network/firewall	Check public URL accessibility

Provider Comparison

Feature	Twilio	Telnyx	Plivo
Auth	Basic (SID:Token)	Bearer token	Basic (ID:Token)
Call Control	TwiML (XML)	Call Control v2 (JSON)	XML
Speech Gather	<Gather>	API command	<GetInput>
Webhook Verify	HMAC-SHA1	Ed25519	HMAC-SHA256
End Call	POST Status=completed	POST hangup action	DELETE
Phone Detection	Auto from account	Manual config	Manual config

How It Works

Outbound Calls — Notify Mode

The agent speaks a message and hangs up. Good for alerts, reminders, and status updates.

User: "Call +1234567890 and tell them the server is back up"
→ Agent uses make_call tool (notify mode)
→ Provider dials the number
→ Person answers → TTS speaks message → Hangup

Outbound Calls — Conversation Mode

Interactive voice exchange with the caller. The assistant listens, thinks, and responds — like a phone conversation.

User: "Call the client and discuss the project timeline"
→ Agent uses make_call tool (conversation mode)
→ Provider dials → Person answers → TTS speaks greeting
→ Person speaks → STT transcribes → LLM responds → TTS speaks back
→ Repeat until either side hangs up

Fast Conversation Path

The webhook conversation path calls the voice-topic model directly:

Caller speaks → Provider STT → Direct LLM call → Provider TTS

This path does not run the full root-agent tool loop. Latency depends on the provider, model, and network. The model assigned to the voice topic is called directly with the Communicator expert’s system prompt plus voice-specific instructions.

Tool Actions

The voice tool currently exposes two handlers:

Action	Description	Permission
make_call	Start a call (notify or conversation mode)	initiate_call permission, default ASK
end_call	Hang up an active call	Default ALLOW; rules can override

Tool exposure depends on the current role and configuration. Unattended ASK calls are rejected; they cannot obtain interactive approval.

Inbound Calls

Inbound calls are disabled by default for security. To enable:

Setting	Value	Description
voice.inboundPolicy	allowlist	Only accept calls from listed numbers
voice.inboundAllowFrom	["+1234567890"]	Allowed caller phone numbers (E.164)

Set voice.inboundPolicy to open to accept calls from any number (not recommended for production).

Configure your provider to send webhooks to:

https://your-public-url/api/voice/webhook/twilio

(Replace twilio with telnyx or plivo as appropriate.)

API Endpoints

Method	Path	Description
POST	/api/voice/transcribe	Transcribe audio with the configured batch STT path
GET	/api/voice/status	Voice subsystem status
POST	/api/voice/webhook/:provider	Telephony webhook
GET	/api/voice/calls	List active calls
GET	/api/voice/telephony/health	Provider health check

Security

• Webhook signature verification for all providers (Twilio HMAC-SHA1, Telnyx Ed25519, Plivo HMAC-SHA256)
• Permission gating — call initiation resolves through the permission engine (default ASK)
• Inbound filtering — allowlist-based caller ID filtering
• Conversation isolation — each call maintains its own conversation context

Distinct voice paths

Batch transcription uses /api/voice/transcribe; synthesis uses /api/voice/speak. Streaming browser voice uses /voice and a different streaming engine path: Whisper.cpp, Mistral, or OpenAI STT, with Piper, Kokoro, Mistral, or OpenAI TTS. Faster-Whisper belongs to the batch path, not the streaming factory. Check readiness for the selected engine before recording.

Carrier webhook conversations use the direct voice-topic model path described above. Twilio also has a media-stream bridge; do not assume equivalent media streaming for Telnyx or Plivo. Server-side push-to-talk relies on local audio tooling such as Linux arecord and is separate from browser microphone capture.