Optimizing Open Models for Long-Horizon Agentic Workloads

Optimizing Open Models for Long-Horizon Agentic Workloads

At a Glance

Item Value
Publisher guide Together AI: models and licenses
Publisher Together AI
Published 2026-11-16
Source type Primary source (the publisher itself)

Values determined by this site’s code when the information was collected. Dates are JST.

Overview

Together AI and Ollama have published explanatory articles on design guidelines for both client (harness) and inference engine sides to improve inference efficiency in long-horizon agentic workloads (such as coding agents) using open-weight models. Because coding agents resend the entire conversation history with each turn, the utilization of prefix caching to cache the beginning of the prompt is essential. This material demonstrates that coordination between prompt structuring on the harness side and request routing or memory management on the inference engine side is crucial for maintaining the cache.

Claims and Rationale

Agentic workloads are characterized by extremely large input sequence lengths (ISL) and short output sequence lengths (OSL). According to the presenter’s explanation, because reprocessing the entire history from the beginning in every turn causes computational complexity to scale quadratically with session length, the use of prefix caching is indispensable.

Numerical comparisons of the idealized computational model presented by the presenters are as follows:

  • Prompt processing volume for 10 turns: Requires 1 + 2 + … + 10 = 55 units of processing without caching, whereas it only requires 10 units with caching.
  • Prompt processing volume for 100 turns: Requires 5,050 units of processing without caching, whereas it only requires 100 units with caching.

Prefix caching is a mechanism that retains tokens matching from the beginning of the prompt and recalculates all processing after the point where the first differing token appears. Therefore, if even a single character changes near the front of the prompt, the cache for the entire past session from that point onward is invalidated. The presenter argues that measures to prevent cache collapse are necessary on both the harness and engine sides.

Claims and Contributing Factors on the Harness (Client) Side

On the harness side, a design that does not alter the beginning portion of the prompt is required. The material lists the following factors and countermeasures:

  • Dynamic data within system prompts: Including timestamps, git status, current directories, or other items that change turn-by-turn in the system prompt causes the cache to be discarded every turn. It is recommended to write these fixed just once on the user task message side.
  • Modifying inference effort settings: In models such as GLM-5.3-Flash, inference effort settings are written in the system block at the beginning of the prompt. Changing the effort midway through a session causes the entire prefix to be recalculated, so it is recommended to leave it at the initial setting or start a new session if changes are needed.
  • Loading position of tool definitions: Inserting tool definitions additionally loaded via tool search functions into the tool list at the beginning of the prompt results in the recalculation of all subsequent history. While appending them to the end is desirable, it depends on the model’s chat template support status.
  • Truncation of tool results: When passing large file read results or the like to the model in a trimmed format, failing to explicitly state that a part was omitted (e.g., that it is lines 1 to 100 out of 640) can cause the model to mistakenly edit unverified code.
  • Sub-agent history inheritance: When utilizing sub-agents in open models such as Kimi K3, GLM-5.3, and GLM-5.3-Flash, inheriting the general history of the parent session allows the prefix to be shared, but memory consumption and attention costs increase.

Claims and Optimization Techniques on the Inference Engine Side

On the inference engine side, it is necessary to maintain once-generated caches and route subsequent requests to the correct node.

  • Cache-aware routing: Because the annual rental cost of a single GPU reaches around $50k, efficient resource utilization is demanded. To route requests of the same session to the same node holding the cache, the use of prompt_cache_key and hashing of prompt prefixes are employed. However, if the target node is under heavy load, load-aware routing—where recalculating on another node is faster than waiting—is sometimes used.
  • Memory management and offloading: While an agent waits for tool execution to complete, techniques are used to offload caches on GPU memory to CPU memory and write them back in the next turn. Additionally, block-based memory allocation via PagedAttention and batch reconstruction via continuous batching are utilized.
  • Separation of prefill and decode: By executing the compute-intensive prefill phase and the memory bandwidth-limited decode phase in separate clusters and transferring cache data via RDMA, long-prompt processing is prevented from delaying other streaming processes.

Prerequisites

The prerequisites for the insights and techniques explained in this material, as well as the targeted model and software environments, are as follows:

  • Target models: Open models such as DeepSeek-V4.1-Flash, GLM-5.3, GLM-5.3-Flash, and Kimi K3 are cited.
  • Client and harness environments: It assumes an environment where various harnesses such as Claude Code, Codex, and Pi are connected to open models using the ollama launch command provided by Ollama.
  • Inference engine configurations and technologies:
  • PagedAttention is used as a cache memory allocation technique. – Continuous batching is adopted for constructing batch processing. – Cache offload processing to CPU memory is incorporated to avoid GPU memory pressure during idle times. – A multi-cluster configuration is assumed, where prefill and decode processes are separated into different clusters, and cache data is transferred between them via RDMA. – Mechanisms for node selection include the assignment of prompt_cache_key, prompt prefix hashing, and load-aware routing.

What Can Be Reproduced Locally

The scope that readers can actually try in their own or operating environments, along with limitations in reproducibility, is as follows:

  • Harness integration via Ollama: By running the ollama launch command distributed by Ollama, existing harness tools such as Claude Code, Codex, and Pi can be connected to open models like DeepSeek-V4.1-Flash and GLM-5.3, allowing them to operate while maintaining the operational workflow.
  • Improvements to prompt structure on the harness side:
  • Moving dynamic values within system prompts (such as timestamps and current directories) to the fixed task message side. – Maintaining inference effort setting values without modification midway through a session, keeping them at their initial settings. – Adding annotations regarding the total line count and extraction range when passing truncated tool execution results.

These prompt design techniques can be directly incorporated into the user’s own client implementation and prompt construction logic.

On the other hand, the specific implementation code and construction procedures for the inference engine backend infrastructure provided by Together AI (such as prefill-decode separated cluster environments, RDMA transfer, CPU memory offload mechanisms, and load control-compatible routing programs) are not publicly available. Therefore, the complete behavior including the engine-side system configuration cannot be reproduced solely in a local environment on hand.

What the Material Does Not Cover

In evaluating the presenter’s claims objectively, the following information is not stated in the material:

  • Quantitative benchmark and actual cost data: Actual measurement data regarding specific processing speed comparisons (latency or throughput) or the reduction scale of API usage costs when performing cache maintenance and offload processing is not shown.
  • Inference engine public code and procedures: Specific source code or sample configuration files for realizing backend inter-node routing or CPU memory offload processing are not posted.
  • Details on loading tool definitions per model: A specific compatibility list regarding how correctly late loading—inserting tool definitions at the end of the prompt—works in each model such as GLM-5.3 or DeepSeek-V4.1-Flash is not presented.
  • Effects in single-GPU or local PC environments: Quantitative behavior and constraints when performing similar optimizations in an individual’s single-GPU environment or general local inference environment, rather than a datacenter-scale distributed node configuration, are not touched upon.

Related Articles

What to Read Next

Sources