Together AI Announces Canary Rollouts for Zero-Downtime Updates

September 23, 2026

At a Glance

Values determined by this site’s code at collection time. Dates are JST.

Overview

Together AI has released “Canary Rollouts," a phased update methodology for zero-downtime model updates in production. This method gradually migrates traffic from a current model (Source) to a new checkpoint or a new model family (Target) according to defined steps.

Claims and Evidence

Together AI claims that by using Canary Rollouts, users can automatically detect performance degradation or errors that may occur when transitioning to a new model through automated metric checks (Metric Gates), providing a mechanism to safely roll back.

In demonstrations by the presenters, when conducting a Canary Rollout from Qwen2.5-7B to Qwen3.5-9B, the gate detected a 137% regression in p95 latency at 10% traffic. At this point, the system automatically entered the “SYSTEM_PAUSED" state, and it is reported that users could cancel the rollout and execute it in reverse to return to the original state without failing live requests.

A comparison of the three strategies provided is as follows:

Prerequisites

The conditions for this method and feature to operate are as follows:

• Target Software: Together AI platform, and the together Python package (version 2.34.0 or later)
• Operation Interface: CLI (tg command), REST API, or console
• Available Metrics: Three types: router_error_rate, router_latency, and inflight_requests
• Metric Constraints:
  * router_latency gate evaluation requires at least 20 requests within the window * p99 evaluation requires at least 100 requests within the window * router_error_rate and inflight_requests require at least 1 data point within the window

Reproducibility

Users utilizing the Together AI platform can implement and control Canary Rollouts via the following methods:

Operation via CLI

You can use the tg command to create, start, monitor, and control rollouts. For example, you can start a 3-step Canary Rollout (10%, 50%, 100%) while monitoring for p95 latency regression with the following command:

# Create and start rollout simultaneously
tg beta endpoints rollout $TARGET_DEPLOYMENT_ID \
--source $SOURCE_DEPLOYMENT_ID \
--canary \
--steps 10,50,100 \
--interval 600s \
--metric router_latency --metric-stat p95 \
--metric-max-regression 10 --metric-direction higher-is-worse \
--metric-window 300s

You can also check the status of the rollout, or pause (--pause), resume (--resume), promote (--promote), and cancel (--cancel) using the following commands:

# Check rollout status
tg beta endpoints get $ROLLOUT_ID

# Cancel rollout
tg beta endpoints rollout $ENDPOINT_ID --cancel --reason "latency regression on target"

Operation via Python SDK

You can control rollouts programmatically using the together package (2.34.0 or later).

from together import Together
client = Together()

rollout = client.beta.endpoints.rollouts.create(
    endpoint_id=ENDPOINT_ID,
    project_id=PROJECT_ID,
    source_deployment_id=SOURCE_DEPLOYMENT_ID,
    target_deployment_id=TARGET_DEPLOYMENT_ID,
    canary={"steps": [{"traffic": 10}, {"traffic": 50}, {"traffic": 100}], "step_interval": "600s"},
    metrics=[{
        "name": "router_latency", 
        "stat": "METRIC_STAT_TYPE_PERCENTILE", 
        "percentile": 95,
        "regression_check": {
            "direction": "REGRESSION_DIRECTION_HIGHER_IS_WORSE", 
            "max_regression_percent": 10
        },
        "window": "300s",
    }],
)
client.beta.endpoints.rollouts.start(rollout.id, project_id=PROJECT_ID, endpoint_id=ENDPOINT_ID)

Operation via REST API

Through the API, you can create and start rollouts with a POST request, and retrieve status with a GET request.

Related Articles

• Scaling Coding Agent Traffic with GLM-5.2 and Dedicated Inference
• Together AI Explains Open-Source AI Stack and MIGHT Stack

Sources

• Canary rollouts: upgrade models in production without downtime