Running Open Models Efficiently for Long-Horizon Agents

At a Glance
| Item | Value |
|---|---|
| Publisher | Ollama |
| Published | 2026-11-16 |
| Source type | Primary source (the publisher itself) |
Values determined by this site’s code when the information was collected. Dates are JST.
Overview
Ollama and Together AI have reported on the technical challenges and solutions for running open-weight models efficiently in agentic workloads. In particular, workloads such as coding agents with long conversation histories feature extremely long input sequence lengths (ISL) and short output sequence lengths (OSL). The report highlights the importance of prefix caching to handle this, along with implementation precautions to avoid invalidating the cache.
Claims and Rationale
The material argues that prefix caching can dramatically reduce computational costs for the property where inputs accumulate as agent execution turns progress.
Differences in Computational Amount Due to Caching
Without caching, all past inputs must be recalculated with every advancing turn. According to the presenter’s explanation, the increase in computational amount is as follows:
- Without caching: Processing $1 + 2 + \dots + 10 = 55$ units of prompts in 10 turns. For 100 turns, it processes 5,050 units.
- With caching: Whether in 10 turns or 100 turns, only the new parts added in each turn are processed, requiring only 100 units of processing even for 100 turns.
Thus, without caching, the workload increases in proportion to the square of the session length.
Design Guidelines for Maintaining the Cache
Prefix caching works by matching from the beginning of the prompt up to the “first differing token." Therefore, if even a single change occurs anywhere in the middle of the prompt, all subsequent caches are invalidated. To prevent this, the following technical approaches are recommended:
- Tool Loading Timing: Placing tool definitions at the beginning of the prompt (before the conversation) causes the beginning of the prompt to change every time a tool is added, destroying the cache. To avoid this, it is effective to place tool definitions at the end of the prompt (where new content is added). However, this depends on model specifications, as chat templates for some models may not expect tool definitions in the middle of the conversation.
- System Prompt Fixation: Including information that changes per turn, such as timestamps, git status, and working directories, in the system prompt destroys the cache every turn. These details should be included in a task message described only once, rather than in the system prompt.
- Inference Setting (Reasoning Effort) Fixation: In models such as GLM-5.3-Flash, the reasoning effort is written into the initial system block, so changing this setting during execution causes differences near the beginning of the prompt, triggering reprocessing of the entire session.
- Engine Routing and Memory Management: On the inference engine side, correctly routing requests to specific nodes holding the cache and offloading idle caches from GPU memory to CPU memory (utilizing PagedAttention, etc.) are essential to maintaining agent response speeds.
Prerequisites
The conditions under which the technical discussions and recommendations mentioned in this material apply are as follows:
- Target Models: DeepSeek-V4.1-Flash, GLM-5.3, GLM-5.3-Flash, Kimi K3, and others.
- Workload Characteristics: Agentic tasks (such as coding agents) with very long input sequence lengths (ISL) and short output sequence lengths (OSL).
- Infrastructure Configuration: Inference engines supporting prefix caching, and environments with nodes or datacenters holding the cache.
- Software and Technologies: Ollama (and connecting various harnesses via the
ollama launchcommand), PagedAttention, continuous batching, and prefill/decode decoupled clusters using RDMA.
What Can Be Reproduced Locally
The following content is presented as the scope that readers can try and implement in their own environments:
- Connecting Harnesses Using Ollama: By using the
ollama launchcommand, existing harnesses such as Claude Code, Codex, and Pi can be connected to open-weight models like DeepSeek-V4.1-Flash and GLM-5.3 while maintaining workflows. - Cache Optimization via Prompt Design:
- Designing the system prompt to be fixed, including dynamic information (such as timestamps and working directories) in task messages. – Configuring tool definitions to be placed at the end of the prompt (limited to cases where the model’s chat template supports it). – Operating with reasoning effort settings fixed at the start of the session.
- Controlling Tool Outputs: Implementing information in the output such as “returned up to line X out of total Y lines" so that the model notices missing information when tools return massive amounts of data.
What the Material Does Not Cover
To evaluate the claims, the following information is not included in the material:
- Specific Benchmark Scores: While theoretical figures are shown for computational reduction through caching, specific measured values for actual inference speed (tokens per second) or latency are not stated.
- Cost Details: Although GPU rental costs are mentioned, specific monetary cost comparisons for using prefix caching, cache offloading, or cache misses occurring are not shown.
- Performance on Specific Hardware: Cluster configurations using RDMA on the inference engine side are touched upon, but detailed specs such as specific GPU models and network bandwidth remain unknown.
- Feasibility of Placing Tools at the End Across All Models: Although placing tool definitions at the end is recommended, a specific list of which models support this method and to what extent is not provided.
Related Articles
- Cloudflare’s Clef Multimodal Decision Model Now on Ollama
- Ollama Announces Integration with Fly.io Sprites
- DeepSeek-V4.1-Flash-1017 Vision-Language Model: ~570GB Memory
What to Read Next
- Engines mentioned in this article → Ollama

