Model Management

Octipus’s model management system provides a unified interface for working with multiple LLM providers, with configured routing, path-specific fallback, usage estimates, and health monitoring.

Provider Types

Direct Providers

Native integrations that communicate directly with the provider API:

Provider	Description
Ollama	Local LLM inference — auto-detected from OLLAMA_URL
OpenAI	OpenAI API — API key stored in vault
Anthropic	Anthropic API — API key stored in vault
Gemini	Google Gemini API — API key stored in vault
OpenRouter	Access models through a shared upstream API
DeepSeek, Grok, Mistral, Moonshot, Zai	Additional direct API adapters
Vertex	Google Vertex AI with service-account authentication
Voyage	Embeddings adapter

CLI Providers

Run models through their vendor CLI as a subprocess. Output handling depends on the execution path and adapter; Antigravity and Vibe buffer their results.

Provider	Description
Claude Code	Anthropic’s Claude via the Claude Code CLI
Antigravity	The agy subprocess adapter
Codex CLI	OpenAI’s Codex via the Codex CLI
Mistral Vibe	The vibe subprocess adapter
Claude Code (z.ai GLM)	claude pointed at z.ai with vendor credentials
Claude Code (Moonshot Kimi)	claude pointed at Moonshot with vendor credentials

These are the six configurations in the CLI registry, using four binaries. Detection checks whether the executable is available; credentials and model access still need verification.

Caution

Billing and quota behavior is vendor-specific and changes over time. Most configurations use the installed CLI’s authentication; the GLM and Kimi adapters inject credentials from environment variables or the system vault. Plan entitlements and billing remain vendor-managed. See CLI Providers for per-vendor flags, auth, model selection, and anti-patterns.

LiteLLM Proxy

An optional OpenAI-compatible proxy for models configured through LiteLLM. Upstream support depends on your proxy version and configuration.

LITELLM_URL=http://localhost:4000

Provider Router

Provider selection matches the configured model to an adapter. Adapter lookup order is not an ordered list of replacement models: an OpenAI error does not mean the next request automatically goes to Anthropic. Child execution selects an explicitly configured executor, then expert model, then primary model where applicable. See Routing for those boundaries.

Topic Routing

Models must be explicitly assigned to topics. When an agent is spawned for a role, it uses the model assigned to that role’s topic.

Concept	Description
Topic	A category like coding, research, general
Primary	The preferred model for a topic
Backup	The fallback model if the primary is unavailable
Unbound topic	If no model is assigned, the agent fails to spawn with a clear error message — there is no silent fallback to the default model

Configure topic routing through the Models page in the web UI or the API. Every role/topic combination you use must have at least a primary model assigned.

Live model discovery

Octipus discovers available models directly from each provider’s list endpoint. There is no static “recommended models” file in the repo — what you see in the picker is whatever the vendor returned on the most recent fetch, run through deterministic curation rules.

Provider	Endpoint	Discovery file
OpenAI	GET /v1/models	src/models/providers/discovery/openai.ts
Anthropic	GET /v1/models (with anthropic-version)	src/models/providers/discovery/anthropic.ts
Google Gemini	GET /v1beta/models	src/models/providers/discovery/gemini.ts
OpenRouter	GET /api/v1/models	src/models/providers/discovery/openrouter.ts
Ollama	GET /api/tags	src/models/providers/discovery/ollama.ts

Curation rules

Each fresh fetch is filtered and sorted by src/models/providers/discovery/curation.ts:

• Recency window — drop models older than ~18 months (when the API exposes a date)
• Capability gate — drop models the API marks as non-tool-capable (OpenRouter); drop embedding/Whisper/TTS/DALL·E/Imagen/Veo/AQA by id pattern
• Family deduplication — collapse dated snapshots (claude-sonnet-4-5-20250929) to their alias (claude-sonnet-4-5) when the alias is also returned
• Preview filter — hide preview/experimental/dated unless ?preview=true is passed
• Tier inference — group results into flagship / balanced / reasoning / cheap from id heuristics
• Sort — flagship → balanced → reasoning → cheap, then createdAt desc

Cached for 6h using the database-backed cache; stale-while-revalidate on errors. Force a refresh with ?refresh=true.

Custom Providers

For endpoints that aren’t backed by a first-party provider class, Octipus offers three custom-provider flavors — pick the one that matches the upstream wire format:

Flavor	provider value	Wire format	Use for
Custom OpenAI-compatible	custom-openai	OpenAI /v1/chat/completions	vLLM, Together, Groq, Fireworks, DeepInfra, internal OpenAI-shaped proxies
Custom Gemini-compatible	custom-gemini	Native Google Gemini (candidates[].content.parts[])	Native Gemini-shaped upstreams
Custom Anthropic-compatible	custom-anthropic	Native Messages	Anthropic-shaped upstreams

Configuration is per model, not per provider — the endpoint URL and key reference live on each model row, so you can register several different upstreams side by side. Each model carries its own apiKeyRef (a vault entry name, or an env:VAR_NAME reference), so there is no single shared key.

1. Secrets page → add a vault entry (e.g. together_api_key) with the upstream’s bearer token. (Or skip this and reference an env var directly with apiKeyRef: 'env:TOGETHER_API_KEY'.)
2. Models page → Add Model with:
   • Provider: custom-openai or custom-gemini or custom-anthropic
   • Endpoint URL: the base URL (no trailing slash) — include /v1 if an OpenAI-compatible upstream expects it (e.g. https://api.together.xyz/v1, http://my-vllm:8000/v1)
   • Model ID: the model name the upstream uses (e.g. meta-llama/Llama-3.3-70B-Instruct-Turbo)
   • API key reference: the vault entry name or env:VAR_NAME for this model
   • Auth scheme: bearer (default), header (custom header name), or query (query param)

The Test button validates connectivity against the configured endpoint and auth scheme.

See Custom Providers for the full schema — auth schemes, apiKeyRef resolution order, native request formats, and tool-calling support.

Model Registry

All model configurations are stored in the database (not environment variables):

• Default model: One model is marked as the default for unrouted messages
• Per-model settings: Enable/disable, provider, topic roles, custom parameters
• Extra body parameters: Per-model custom parameters via metadata.extraBody (e.g., { think: false } for Qwen3)

Model Test Endpoint

Before registering a model, you can validate connectivity:

POST /api/models/test

The test uses the selected model/provider configuration. Check the returned result before assigning the model to a topic.

Cost Tracking

The system tracks per-model token costs:

• Input tokens: Tokens sent to the model
• Output tokens: Tokens generated by the model
• Cost calculation: Based on per-model pricing configuration
• Aggregation: Costs aggregated by model, time period, and user

Quota Tracking

Database-backed usage tracking helps monitor configured quotas; it does not guarantee compliance with upstream limits or exact billing:

• Daily quotas: Track usage per model per day
• Auto-clearing: Quotas reset automatically at the start of each day
• Exhaustion detection: Exhausted quotas can block requests; fallback requires a supported execution path and configured alternative

Health Checks

Periodic health monitoring for all configured providers:

• Latency measurement: Tracks response times for each provider
• Availability status: Marks providers as healthy or unhealthy
• Auto-recovery: Unhealthy providers are re-checked periodically and restored when available

Access health status via:

GET /api/models/health

Per-Provider Rate Limiting

Each provider has independent rate limiting with adaptive concurrency and circuit breaker protection. These controls reduce request pressure; upstream rate limits and transient errors can still occur.

Supported Providers

Rate limiting is configured per-provider for:

• Ollama — local inference (concurrency limited by GPU memory)
• OpenAI — cloud API (tokens-per-minute and requests-per-minute)
• Anthropic — cloud API (requests-per-minute)
• Gemini — cloud API (requests-per-minute)
• DeepSeek — cloud API (requests-per-minute)
• OpenRouter — multi-model proxy (credit-based limits)
• LiteLLM — proxy (inherits downstream limits)

Adaptive Concurrency

The rate limiter dynamically adjusts the number of concurrent requests based on provider response times:

• Scale up: When responses are fast, concurrency increases to maximize throughput
• Scale down: When latency rises or errors occur, concurrency decreases to reduce pressure
• Per-provider: Each provider has its own concurrency window

Circuit Breaker

When a provider experiences repeated failures, the circuit breaker trips to prevent cascading issues:

State	Behavior
Closed	Normal operation — requests flow through
Open	Provider is failing — requests are rejected while the circuit is open
Half-Open	After a cooldown period, a single test request is sent to check recovery

The circuit breaker transitions back to Closed once the provider responds successfully in the half-open state. Fallback is execution-path dependent; a circuit breaker does not itself select a healthy replacement model.

Configuration

Environment Variables

LITELLM_URL=http://localhost:4000      # LiteLLM proxy (optional)
OLLAMA_URL=http://localhost:11434      # Local Ollama (optional)
OPENROUTER_API_KEY=sk-or-...           # OpenRouter API key (optional)

OpenRouter Setup

OpenRouter provides access to 200+ models from multiple providers through a single API key:

1. Create an account at openrouter.ai and add credits
2. Store your API key in the vault as openrouter_api_key or set OPENROUTER_API_KEY
3. Register models with provider: 'openrouter' and modelId in provider/model format (e.g., minimax/minimax-01, nvidia/llama-3.1-nemotron-ultra-253b-v1)
4. Use the OpenRouter model search in the web UI to browse and register available models

Caution

OpenRouter model IDs must use the provider/model format (e.g., openai/gpt-4o). The system validates this format during registration.

Note

Provider API keys (OpenAI, Anthropic, Gemini) are stored in the encrypted vault, not in environment variables. Configure them through the Settings page.

Useful API Endpoints

Method	Endpoint	Description
GET	/api/models	List all models
POST	/api/models	Register a new model
POST	/api/models/test	Test model connectivity
POST	/api/models/:name/default	Set as default model
GET	/api/topics	View topic routing
GET	/api/models/health	Provider health status
GET	/api/models/cli/status	CLI tool availability
GET	/api/models/cli/quota	CLI quota status

Environment variables bootstrap relevant settings on first setup; persisted database configuration and vault values govern later runs. Confirm effective settings instead of assuming an environment edit overrides an existing record.