Skip to content
Octipus Logo Octipus

Running Octipus on Ollama

Octipus treats Ollama as a first-class local provider: self-hosted, no API key, runs on your own CPU/GPU. This doc covers what Octipus expects from an Ollama model — size, tool support, context length — and the optimizations (lazy tool discovery, iGPU) that make a swarm usable on consumer hardware. Lazy tool discovery is gated on model capability, not provider — see below — but it matters most here, where a local iGPU re-runs prefill on every request.

Related: Small / Local Models (single-model setup and what degrades) and Custom Providers (OpenAI-compatible endpoints). This doc is the Ollama-specific companion to both.

TL;DR

Section titled “TL;DR”
QuestionAnswer
Minimum viable model for the full swarm (tools + multi-step)~7–8B, 4-bit, supportsTools: true (e.g. qwen3:8b)
“Small model” threshold (triggers prompt/tool trimming)< 10B params (agent.smallModelMaxParams)
Tool callingNative Ollama tool format; thinking must stay ON for agent workers
Context Octipus packs before compactionagent.contextWindowSize, default 32000 tokens
Context Ollama actually allocatesnum_ctx / OLLAMA_CONTEXT_LENGTH — set this yourself (default 4096!)
iGPU (AMD 890M etc.)OLLAMA_IGPU_ENABLE=1 or it silently drops to CPU

Adding Ollama models

Section titled “Adding Ollama models”

Add models in the Models page (or seed via API) with provider: 'ollama'. Octipus talks to Ollama’s native /api/chat endpoint (not the /v1 shim — the native format gives reliable tool calls and JSON mode). Default endpoint is http://localhost:11434; override per-model via the endpoint field or globally with OLLAMA_URL.

Pull the weights first (ollama pull qwen3:8b), or use the hardware-aware onboarding (octi capabilities / hwfit), which reads your VRAM and recommends pullable tags from a curated catalog (src/capabilities/hwfit/catalog.json).

Model size — and the small-model threshold

Section titled “Model size — and the small-model threshold”

Octipus routes by parameter count, derived from metadata.paramCount or parsed from the model tag (qwen3:8b → 8B; qwen3:30b-a3b MoE → counted by total). See deriveParamCount (src/core/agent/model-selector.ts).

A model under agent.smallModelMaxParams (default 10B) is treated as small and gets:

  • Prompt trimming — the expert scaffold, deliverable templates, and success metrics are dropped (a weak model can’t drive a multi-section prompt).
  • Tool cap — the surface is capped to smallModelMaxTools (default 7) tool groups, priority-ordered, so core tools survive and the long tail is dropped wholesale.
  • No lazy tool discovery — small models chain multi-step discovery poorly, so they keep the capped full-schema path (see below).

Rough capability tiers (4-bit quant, consumer GPU):

SizeUseNotes
≤ 1.7Btoy / classification onlyWon’t reliably drive tools or multi-step tasks
4Bcasual chat, light single-toolSmall-tier: trimmed prompt + capped tools
7–8Bpractical floor for the swarmReliable tool calls; good default for ~8GB GPUs
14–32Bhigher-fidelity research/codingNeeds 12–24GB VRAM (or MoE like qwen3:30b-a3b)

House rule #2: never hardcode a model. Bind topics to models and let ModelRegistry resolve. Size-based behavior above is automatic.

Quantization

Section titled “Quantization”

Catalog tags default to q4_K_M (4-bit) — the sweet spot for quality vs VRAM. Going below q4 (q3/q2) degrades tool-call reliability noticeably; prefer a smaller model at q4 over a larger model at q2. The vramHintMB in the catalog is an offline fallback estimate only — real weight size is fetched live from the Ollama registry manifest at recommend-time.

Tool use

Section titled “Tool use”

Ollama models tool-call via Ollama’s native tool format (Octipus maps the OpenAI-style tool schema in ollama-provider.ts). Two things to know:

  1. supportsTools gates everything. Ollama models default to supportsTools: true; set it to false for a model that can’t tool-call and it will be given no tools at all (and never routed for tool work). Not every small Ollama model tool-calls well — verify before relying on it.
  2. Thinking stays ON for agent workers. Empirically, Ollama models with think:false emit malformed tool-call JSON that Ollama’s Go-side parser rejects ("Value looks like object, but can't find closing '}'"). Octipus therefore strips a think:false from extraBody for tool-calling workers (agent-worker.ts). The reasoning tokens are stripped before delivery, so the delivery path attempts to remove them. This mitigation does not guarantee valid tool calls for every model/version. Casual chat (no tools) keeps the model’s configured think setting.

Lazy tool discovery

Section titled “Lazy tool discovery”

Every advertised tool’s full JSON schema is sent on every request. For a research worker that was ~12k tokens of schemas against a ~2.4k system prompt — and on a local iGPU that prefill re-runs every request (no cross-request server-side prompt caching), which took minutes and timed out the client. A benchmark later showed the same cost hits remote providers too: the root’s standing prompt measured ~22k tokens, 73% of it tool JSON schema, billed in full on every fresh session since a cache prefix only helps within one.

Lazy tool discovery advertises only a small core set with full schema, plus two meta-tools — list_tools (names + one-line descriptions, no params) and describe_tool (full schema for one tool). The long tail stays registered and callable by name; the model fetches a schema on demand (one extra round-trip on first use).

It is gated on the model’s capability, not its provider — non-small + supportsTools + a role that opts in via coreToolIds (see src/core/agent/lazy-tools.ts):

  • Small models → keep the capped full-schema path (they chain multi-step discovery poorly).
  • A role with no coreToolIds → byte-for-byte unchanged behavior.
  • agent.lazyToolDiscovery (default true) is the operator override — set it false to force full-schema advertisement everywhere, e.g. for a provider that measurably tool-calls worse without the full block.

A role opts in by setting coreToolIds ⊆ toolIds in its config. Example (roles/research/config.ts): coreToolIds: ['websearch', 'knowledge'] — the hot path stays core; filesystem/profiles/artifacts/task_state become long tail; mcp stays core automatically (it’s its own discovery surface).

Context length — two different knobs

Section titled “Context length — two different knobs”

This trips people up because there are two context settings:

  1. agent.contextWindowSize (Octipus, default 32000). How much conversation history Octipus packs before it compacts/summarizes. Octipus side only.
  2. num_ctx / OLLAMA_CONTEXT_LENGTH (Ollama). How large a KV context Ollama actually allocates for the model. Ollama’s default is only 4096 tokens — far smaller than Octipus’s 32000. Octipus does not force num_ctx per request, so if you leave Ollama at its default, long prompts (system prompt + tools + history) get silently truncated and tool calls degrade.

Set Ollama’s context yourself to match your work and VRAM:

Terminal window
# Global, applies to all models (Ollama ≥ 0.6):
OLLAMA_CONTEXT_LENGTH=32768 ollama serve

or bake it into a Modelfile (PARAMETER num_ctx 32768). Minimum guidance:

WorkloadRecommended num_ctx
Casual chat, no tools8k
Tool-using worker (full schema)16k+ — the tool block alone can be several k
Tool-using worker (lazy discovery)8–16k (smaller payload)
Research / long synthesis32k+

Bigger context costs VRAM (the KV cache grows with num_ctx), so size it to the job. Catalog models advertise large native windows (Qwen3 ~40960, Gemma3 32768), but you still have to allocate it via num_ctx.

GPU / iGPU notes

Section titled “GPU / iGPU notes”

  • AMD integrated GPUs (e.g. Radeon 890M). Ollama 0.30.x drops the 890M iGPU by default and silently falls back to CPU → slow prefill and timeouts. Fix with OLLAMA_IGPU_ENABLE=1 (and keep any HSA_OVERRIDE_GFX_VERSION gfx override you already had). In Docker Compose, env changes need up -d to recreate the container — a plain restart won’t pick them up.

  • VRAM spill. On a machine with a discrete GPU, models larger than VRAM spill to system RAM over PCIe and run much slower. Use the hwfit recommender to pick a tag that fits, or a mixture-of-experts model (qwen3.5:35b-a3b: 35B total, ~3B active) for capability per gigabyte.

  • Unified memory (ollama.unifiedMemory). On an APU or an Apple Silicon machine there is no spill to speak of: the reported VRAM is a firmware carve-out from the same DRAM the GPU reaches through its GTT aperture, at the same speed. Scoring against the carve-out is then measuring a boundary that does not exist — a Radeon 890M reporting 16 GB alongside 39 GB of GTT was told that everything above 16 GB “spills into RAM, slower”, which hid the best model the machine could run.

    Apple Silicon is detected automatically. AMD is not detectable on Linux — an APU and a discrete card expose the same sysfs surface, with no mem_info_vram_vendor on current kernels and a GTT aperture that looks the same on both — so set ollama.unifiedMemory (or OLLAMA_UNIFIED_MEMORY=true, or octi models recommend --unified for a single run) and the recommender scores against VRAM + GTT instead.

    The pool is bounded by GTT, not by installed RAM: 16 GB VRAM + 39 GB GTT on a 96 GB machine gives a 55 GB budget, because the amdgpu driver sizes the aperture at roughly half of system memory by default. Raise it with the amdgpu.gttsize kernel parameter (in MB) if you want more of the pool available to the GPU.

Section titled “Recommended models (from the curated catalog)”

Current-gen, pullable, tool-capable defaults (q4_K_M):

TagParams~VRAMGood for
qwen3:8b-q4_K_M8B~5.2GBDefault all-round; ~8GB GPUs
qwen3:4b-q4_K_M4B~2.9GBSmall chat/reasoning (small-tier)
qwen3:14b-q4_K_M14B~9.3GBHigher-fidelity general/reasoning
qwen3:30b-a3b-q4_K_M30B MoE (~3B active)~18.5GBCapable yet fast
qwen3-coder / qwen2.5-coder——Coding role
qwen3-embedding——Embeddings (RAG)

See src/capabilities/hwfit/catalog.json for the full list and the live-sizing behavior.

Troubleshooting

Section titled “Troubleshooting”
SymptomLikely causeFix
Requests time out on first tool turniGPU dropped to CPU, or num_ctx too small for the tool blockOLLAMA_IGPU_ENABLE=1; raise OLLAMA_CONTEXT_LENGTH
Malformed tool-call JSON / parser errorsthink:false on a tool-calling modelOctipus strips it for workers; check the model’s extraBody if you see it in chat
Model never gets toolssupportsTools: false, or routed as small + capped outCheck the model’s supportsTools flag and size tier
Prompt seems truncatedOllama at default 4096 num_ctxSet OLLAMA_CONTEXT_LENGTH
Tool payload huge / slow prefill on localFull-schema advertisementOpt the role into lazy discovery (coreToolIds)

Edit this page in the repository.