Scaling Coding Agent Traffic with GLM-5.2 and Dedicated Inference
At a Glance
| Item | Value |
|---|---|
| Publisher | Together AI |
| Published | 2026-09-18 |
| Source type | Primary source (the publisher itself) |
Values determined by this site’s code at collection time. Dates are JST.
Overview
Together AI published a case study detailing how a global financial institution ran the Mixture-of-Experts (MoE) model “GLM-5.2" on its Dedicated Model Inference (DMI) platform as an inference backend for internal coding agents. To handle sudden burst loads during business hours caused by a large number of engineers, the setup employed a multi-replica configuration using 56 NVIDIA B200 GPUs (with a 256K context length) to ensure concurrency, addressing queueing delays through self-service operations and real-time configuration changes.
Claims and Evidence
The central claim of the presentation is that for inference operations of long-context coding agents, ensuring concurrency capable of withstanding sudden traffic concentration and routing control to prevent prefill processing bottlenecks are more important than simply maximizing token throughput (TPS).
The presentation provides the following measured values for the coding assistant traffic at the target financial institution:
- Input Sequence Length (ISL):
- p50: approx. 81,000 tokens – p90: approx. 163,000 tokens – p95: approx. 178,000 tokens
- Request Frequency (RPS):
- p50: 3 RPS – p90: 6 RPS – p95: 7 RPS
This workload has a “peak load" characteristic where traffic concentrates during engineers’ working hours, and while the number of requests itself is relatively low frequency (low TPS), each request has a massive input prompt, causing concurrency to spike. With conventional static capacity planning, it was difficult to predict bursts when multiple teams increased agent usage simultaneously, leading to exhausted prefill processing capacity and KV cache availability against pre-allocated capacity, resulting in request backlogs (queueing) lasting several minutes.
According to the presenter’s measurements and operational logs, during the operation of GLM-5.2, when the endpoint configuration was changed to reduce the number of replicas while increasing the number of chips per replica, an incident occurred a few days later where prefill capacity reached nearly 100%. At this time, request queue dwell times reached 1 to 3 minutes, and decode throughput collapsed to approximately 5 tokens per second (tokens/sec).
An investigation via the metrics API showed the tracked results of the processing breakdown for a single slow request that took a total of 192 seconds. While this request itself had a prompt length of 250,000 tokens, it was discovered that the majority of the processing time was not due to a compute-bound condition, but rather to queue waiting behind a pending prefill backlog of 2.3 million tokens (2.3M tokens) caused by other requests.
It is reported that live configuration changes were implemented to address this issue without redeploying. Specifically, the default DMI hash-based cache routing policy was changed to a tuned “cache-session-aware routing policy," and the max-inflight-per-worker threshold was expanded, completing recovery on the same day without downtime.
Evidence was also provided regarding the selection of the hardware configuration. During the initial verification stage, GLM 5.1 was used to conduct concurrency tests with 8 to 16 B200 GPUs, and upon meeting the criteria, the system was migrated to the production environment. Subsequently, with the migration to GLM-5.2, a configuration using 56 B200 GPUs (4 B200 GPUs × 14 replicas) at a 256K context length was deployed. Although expansion to a 1M (1 million) context length was considered during the evaluation, a trade-off was confirmed where doubling the context length to 1M reduced the concurrency headroom required during peak loads, leading the company to choose to maintain the 256K and 512K settings.
Prerequisites
Based on the documentation, the conditions under which the configurations and verification results shown in this case study hold are as follows:
- Target Model: GLM-5.2 (Mixture-of-Experts model; GLM 5.1 was used in the preliminary verification phase)
- Target Hardware:
- Production Environment: Total of 56 NVIDIA B200 GPUs (4 B200 GPUs × 14 replicas configuration) – Initial Verification Environment: 8 to 16 B200 GPUs
- Operating Platform and Software:
- Together AI Dedicated Model Inference (DMI) – Endpoint lifecycle control (creation, sizing, scaling policy changes) via API, UI, and CLI, as well as the metrics API
- Configuration Parameters and Setup:
- Context Length: 256K (512K also considered in some evaluations; 1M context length omitted due to concurrency constraints) – Routing Settings: Tuned cache-session-aware routing policy (initial state was hash-based cache-aware-by-hash) – Concurrency Settings: Expanded max-inflight-per-worker threshold
- Traffic Characteristics of Target Workload:
- Input Sequence Length (ISL): p50 is approx. 81K tokens, p90 is approx. 163K tokens, p95 is approx. 178K tokens – Request Frequency (RPS): p50 is 3, p90 is 6, p95 is 7 – Traffic Trends: Sharp burst characteristics concentrated during engineers’ working hours, low TPS with high concurrency
What You Can Reproduce Locally
It is not possible to directly reproduce the operational configuration shown in this material on a personal PC or a typical small-scale on-premise environment.
- The procedures, inference code, and configuration scripts required for reproduction are not publicly available. This document is a case study report on Together AI’s commercial Dedicated Model Inference (DMI) platform and does not include internal implementations of the inference engine or detailed cluster orchestration code.
- The hardware used is limited to extremely large-scale infrastructure utilizing up to 56 datacenter-grade NVIDIA B200 GPUs.
- Elements that readers can reference locally are limited to design insights—such as the fact that concurrency and KV cache management tend to become bottlenecks rather than throughput in long-context coding agents (handling 256K tokens or more)—and descriptions of trade-offs where excessively increasing the context length degrades concurrent execution capacity.
What the Material Does Not Cover
The documentation lacks the following information necessary to objectively evaluate this case study:
- Specific total parameters, active parameters, and the presence or format of weight quantization applied (FP8, FP4, INT4, etc.) for GLM-5.2
- The specific company name or deployment region of the global financial client
- Interconnect specifications between B200 nodes (such as NVLink or network bandwidth) and specific host machine specifications
- Infrastructure costs incurred by using the DMI platform and cost-efficiency data compared to other cloud inference environments or self-hosted environments
- Specific algorithms and implementation details of the cache-session-aware routing policy, as well as specific numerical settings for “max-inflight-per-worker"
- Third-party reproducibility verification or performance comparisons under identical environments with other models (such as open-weight MoE models from other companies)