ByteDance-Seed Releases Cola2 Language Model

ByteDance-Seed Releases Cola2 Language Model

At a Glance

Item Value
Repository ByteDance-Seed/Cola2
Publisher guide ByteDance Seed: models and licenses
Published 2026-11-06
Formats safetensors
Paper arXiv:2610.31305
Source type Primary source (the publisher itself)

Values determined by this site’s code when the information was collected. Dates are JST.

Overview

ByteDance-Seed has released “Cola2", a new language model combining semantic compression via a continuous latent space with discrete text generation. This model features an architectural structure that integrates a dynamic compression VAE, a block-causal Diffusion Transformer (DiT), and a span-causal token decoder with latent feedback.

The underlying research paper for the model is available as “Cola2: Co-Adaptive Hierarchical Language Modeling with Continuous Semantic Compression and Balanced Discrete Generation“. According to the paper’s abstract, Cola2 was developed to achieve co-adaptation between continuous semantic compression and balanced discrete generation, by dynamically grouping text into semantic spans to represent them as compact continuous latent variables, generating latent blocks with a DiT, and then reconstructing them as discrete tokens.

Specifications

  • Architecture: Dynamic compression VAE, block-causal Diffusion Transformer (DiT), and span-causal token decoder with latent feedback (the DiT portion is approximately 4B parameters)

Performance

The post-training evaluation results published by the publisher on the model card and a comparison table with open-weight models of similar scale are shown below. Except for the row of pretraining tokens, the values in the table indicate accuracy (%). In addition, the values for each task of Cola2 are averages over three evaluations with latent noise sampling.

→ Scroll horizontally to see all columns

Task Llama-3-8B Qwen2.5-7B Qwen3-8B-Base LLaDA-8B-Base Cola2
Pretraining tokens 15T+ 18T ~36T 2.3T 346B
RACE 75.77 89.95 90.65 87.40 69.91
NarrativeQA 42.70 32.90 37.30 36.00 32.70
HellaSwag 39.21 75.67 68.13 69.65 41.75
StoryCloze 79.49 95.09 97.44 95.94 80.42
LAMBADA 75.70 71.82 71.20 67.90 63.00
ARC-Easy 90.86 96.00 97.35 93.85 87.62
TruthfulQA 63.04 69.03 72.58 64.87 54.10
MMLU 63.19 70.16 72.31 61.27 51.32
8-task average 66.25 75.08 75.87 72.11 60.10

According to the publisher’s measurement results, despite having a relatively small pretraining dataset size of 346.03B source tokens, Cola2 scores higher in HellaSwag (41.75) and StoryCloze (80.42)—which measure commonsense reasoning ability by choosing natural continuations of sentences—than Llama-3-8B (39.21 and 79.49, respectively), which was trained on more than 43 times the amount of data.

On the other hand, the average score across all 8 tasks (60.10) is the lowest among all compared models. In particular, in items such as MMLU (51.32) which evaluates broad and shallow general knowledge, and TruthfulQA (54.10) and ARC-Easy (87.62) which measure factuality such as resistance to misconceptions, the results fall short of the Qwen series, LLaDA-8B-Base, and Llama-3-8B.

Note that it is explained that evaluation results may vary for each individual run because probabilistic elements are included in the latent variable generation process.

Strengths and Use Cases

Cola2 is provided as a base model rather than an instruction-tuned chat assistant. Therefore, rather than being used directly as a response engine for general conversational applications, it is suited for use cases such as hierarchical language modeling using a continuous latent space, research and experimentation on text generation algorithms based on Diffusion Transformer (DiT), and feature extraction.

According to the model card description, despite a relatively small pretraining dataset size, it shows efficient results in commonsense reasoning tasks such as HellaSwag and StoryCloze, making it suitable for environments to verify the co-adaptation mechanism of continuous semantic compression and discrete generation, and to evaluate the performance of new generation architectures.

Hardware Requirements

Estimated requirements (calculated by Local Model Watch) — 8.8B parameters

Your VRAM Quantization File size Est. memory needed
48GB (RTX 6000 Ada / A6000, etc.) F32 32.7GB 39.2GB

Inference engine support (architecture name matched against each project’s own model registry in its source code, checked 2026-11-17): llama.cpp: not registered, vLLM: not registered, MLX (mlx-lm): not registered. “Not registered" means the name is absent from that registry today, not that the model cannot run.

Memory estimates add a 20% runtime overhead (KV cache, etc.) to the actual size of the distributed files. Actual usage varies with context length, batch size and inference engine. These figures are computed by this site from file sizes, not published by the model’s authors. The listed files are distributed in F32 (32-bit). Loading in a lower precision (BF16 or quantized) needs less memory, so treat this as the size of the files as distributed, not a minimum. This site has not measured the lower-precision figures. Compare with other models in our VRAM quick reference. What the quantization names mean: glossary.

Can You Run It Locally?

Not usable in Ollama, LM Studio and llama.cpp yet.

The publisher ships safetensors only, and llama.cpp’s registry does not list this architecture. llama.cpp would need to add support before these tools can run it. Today it can be run with transformers, using the memory figures in the table above.

Compiled by this site’s code from the published formats, converted builds we have found, and each engine’s own model registry. “Not found" means we have not seen such a build, not that none exists.

Our Own Measurements

Measurements We Did Not Take

This model does not state a license, so we cannot confirm permission to use it. Because this site carries advertising, we did not run the model (no CPU run, answers, quantization comparison, conversion or generation).

Recent Models in the Same Size Class

Models with 4–15B parameters that Local Model Watch covered recently, listed by code from our article log for comparison. VRAM tiers are this site’s estimates; licenses are as stated on the model cards.

→ Scroll horizontally to see all columns

Model Parameters Smallest VRAM tier License Our article
microsoft/UserRepGen-4B 4.5B 4GB mit UserRepGen-4B Text Generation Model: 4GB+ VRAM, GGUF Builds (2026-11-17)
IFM/Jais-3-7B-GGUF 9.0B 8GB apache-2.0 Jais-3-7B-GGUF Arabic and English 7B Reasoning Model: 8GB+ VRAM (2026-11-17)
IFM/Jais-3-3.7B-GGUF 5.1B 4GB apache-2.0 Jais-3-3.7B-GGUF 512K Context Language Model: 4GB+ VRAM (2026-11-17)
ibm-granite/granite-switch-4.1-8b 8.8B 24GB apache-2.0 IBM Releases Granite Switch 4.1: Modular LLMs with aLoRA and Speech (2026-11-17)
ggml-org/NVIDIA-Nemotron-3.5-Bolt-E4B-GGUF 8.0B 12GB apache-2.0 NVIDIA-Nemotron-3.5-Bolt-E4B-GGUF Text Generation Model: 12GB+ VRAM (2026-11-17)

How to Get It

Cola2 model files are released in an 8-part safetensors format and can be obtained from Hugging Face. No special license agreement procedure (gated) is required for downloading.

As execution environments, Python 3.10 or higher, PyTorch 2.2 or higher, Transformers 4.51 or higher and less than 5, safetensors 0.4 or higher, and the accelerate package are required, and a CUDA environment supporting bfloat16 is recommended. In addition, because this model has a unique structure, trust_remote_code=True must be specified when loading from Transformers.

A minimal code example for loading the model from a Python script and performing inference is as follows:

import torch
from transformers import AutoModel, AutoTokenizer

model_id = "ByteDance-Seed/Cola2"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModel.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    low_cpu_mem_usage=True,
).to("cuda").eval()

The following procedure can be used to download the complete set of model files from the command line and verify the file checksums:

hf download ByteDance-Seed/Cola2 --local-dir /path/to/cola2
cd /path/to/cola2
sha256sum -c SHA256SUMS

Note that command-line chat scripts, benchmark execution tools, and various documents are managed in the official GitHub repository.

Related Articles

What to Read Next

Sources