{"id":15301,"date":"2026-11-17T11:03:38","date_gmt":"2026-11-17T02:03:38","guid":{"rendered":"https:\/\/localmodelwatch.tsuchitsuchi.com\/?p=15301"},"modified":"2026-11-17T11:13:12","modified_gmt":"2026-11-17T02:13:12","slug":"bytedance-seed-releases-cola2-language-model","status":"publish","type":"post","link":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/bytedance-seed-releases-cola2-language-model\/","title":{"rendered":"ByteDance-Seed Releases Cola2 Language Model"},"content":{"rendered":"<p><!-- lmw:facts --><\/p>\n<h2>At a Glance<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Item<\/th>\n<th>Value<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Repository<\/td>\n<td><a href=\"https:\/\/huggingface.co\/ByteDance-Seed\/Cola2\">ByteDance-Seed\/Cola2<\/a><\/td>\n<\/tr>\n<tr>\n<td>Publisher guide<\/td>\n<td><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/publisher-bytedance-en\/\">ByteDance Seed: models and licenses<\/a><\/td>\n<\/tr>\n<tr>\n<td>Published<\/td>\n<td>2026-11-06<\/td>\n<\/tr>\n<tr>\n<td>Formats<\/td>\n<td><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/format-safetensors-en\/\">safetensors<\/a><\/td>\n<\/tr>\n<tr>\n<td>Paper<\/td>\n<td><a href=\"https:\/\/arxiv.org\/abs\/2610.31305\">arXiv:2610.31305<\/a><\/td>\n<\/tr>\n<tr>\n<td>Source type<\/td>\n<td>Primary source (the publisher itself)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p><em>Values determined by this site&#8217;s code when the information was collected. Dates are JST.<\/em><\/p>\n<p><!-- \/lmw:facts --><\/p>\n<h2>Overview<\/h2>\n<p>ByteDance-Seed has released &#8220;Cola2&#8221;, a new language model combining semantic compression via a continuous latent space with discrete text generation. This model features an architectural structure that integrates a dynamic compression VAE, a block-causal Diffusion Transformer (DiT), and a span-causal token decoder with latent feedback.<\/p>\n<p>The underlying research paper for the model is available as &#8220;<a href=\"https:\/\/arxiv.org\/abs\/2610.31305\">Cola2: Co-Adaptive Hierarchical Language Modeling with Continuous Semantic Compression and Balanced Discrete Generation<\/a>&#8220;. According to the paper&#8217;s abstract, Cola2 was developed to achieve co-adaptation between continuous semantic compression and balanced discrete generation, by dynamically grouping text into semantic spans to represent them as compact continuous latent variables, generating latent blocks with a DiT, and then reconstructing them as discrete tokens.<\/p>\n<h2>Specifications<\/h2>\n<ul>\n<li>Architecture: Dynamic compression VAE, block-causal Diffusion Transformer (DiT), and span-causal token decoder with latent feedback (the DiT portion is approximately 4B parameters)<\/li>\n<\/ul>\n<h2>Performance<\/h2>\n<p>The post-training evaluation results published by the publisher on the model card and a comparison table with open-weight models of similar scale are shown below. Except for the row of pretraining tokens, the values in the table indicate accuracy (%). In addition, the values for each task of Cola2 are averages over three evaluations with latent noise sampling.<\/p>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<p class=\"lmw-table-hint\" style=\"margin:0 0 4px;font-size:0.85em;opacity:0.7;\">\u2192 Scroll horizontally to see all columns<\/p>\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Task<\/th>\n<th style=\"text-align: right;\">Llama-3-8B<\/th>\n<th style=\"text-align: right;\">Qwen2.5-7B<\/th>\n<th style=\"text-align: right;\">Qwen3-8B-Base<\/th>\n<th style=\"text-align: right;\">LLaDA-8B-Base<\/th>\n<th style=\"text-align: right;\">Cola2<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Pretraining tokens<\/td>\n<td style=\"text-align: right;\">15T+<\/td>\n<td style=\"text-align: right;\">18T<\/td>\n<td style=\"text-align: right;\">~36T<\/td>\n<td style=\"text-align: right;\">2.3T<\/td>\n<td style=\"text-align: right;\"><strong>346B<\/strong><\/td>\n<\/tr>\n<tr>\n<td>RACE<\/td>\n<td style=\"text-align: right;\">75.77<\/td>\n<td style=\"text-align: right;\">89.95<\/td>\n<td style=\"text-align: right;\">90.65<\/td>\n<td style=\"text-align: right;\">87.40<\/td>\n<td style=\"text-align: right;\"><strong>69.91<\/strong><\/td>\n<\/tr>\n<tr>\n<td>NarrativeQA<\/td>\n<td style=\"text-align: right;\">42.70<\/td>\n<td style=\"text-align: right;\">32.90<\/td>\n<td style=\"text-align: right;\">37.30<\/td>\n<td style=\"text-align: right;\">36.00<\/td>\n<td style=\"text-align: right;\"><strong>32.70<\/strong><\/td>\n<\/tr>\n<tr>\n<td>HellaSwag<\/td>\n<td style=\"text-align: right;\">39.21<\/td>\n<td style=\"text-align: right;\">75.67<\/td>\n<td style=\"text-align: right;\">68.13<\/td>\n<td style=\"text-align: right;\">69.65<\/td>\n<td style=\"text-align: right;\"><strong>41.75<\/strong><\/td>\n<\/tr>\n<tr>\n<td>StoryCloze<\/td>\n<td style=\"text-align: right;\">79.49<\/td>\n<td style=\"text-align: right;\">95.09<\/td>\n<td style=\"text-align: right;\">97.44<\/td>\n<td style=\"text-align: right;\">95.94<\/td>\n<td style=\"text-align: right;\"><strong>80.42<\/strong><\/td>\n<\/tr>\n<tr>\n<td>LAMBADA<\/td>\n<td style=\"text-align: right;\">75.70<\/td>\n<td style=\"text-align: right;\">71.82<\/td>\n<td style=\"text-align: right;\">71.20<\/td>\n<td style=\"text-align: right;\">67.90<\/td>\n<td style=\"text-align: right;\"><strong>63.00<\/strong><\/td>\n<\/tr>\n<tr>\n<td>ARC-Easy<\/td>\n<td style=\"text-align: right;\">90.86<\/td>\n<td style=\"text-align: right;\">96.00<\/td>\n<td style=\"text-align: right;\">97.35<\/td>\n<td style=\"text-align: right;\">93.85<\/td>\n<td style=\"text-align: right;\"><strong>87.62<\/strong><\/td>\n<\/tr>\n<tr>\n<td>TruthfulQA<\/td>\n<td style=\"text-align: right;\">63.04<\/td>\n<td style=\"text-align: right;\">69.03<\/td>\n<td style=\"text-align: right;\">72.58<\/td>\n<td style=\"text-align: right;\">64.87<\/td>\n<td style=\"text-align: right;\"><strong>54.10<\/strong><\/td>\n<\/tr>\n<tr>\n<td>MMLU<\/td>\n<td style=\"text-align: right;\">63.19<\/td>\n<td style=\"text-align: right;\">70.16<\/td>\n<td style=\"text-align: right;\">72.31<\/td>\n<td style=\"text-align: right;\">61.27<\/td>\n<td style=\"text-align: right;\"><strong>51.32<\/strong><\/td>\n<\/tr>\n<tr>\n<td><strong>8-task average<\/strong><\/td>\n<td style=\"text-align: right;\"><strong>66.25<\/strong><\/td>\n<td style=\"text-align: right;\"><strong>75.08<\/strong><\/td>\n<td style=\"text-align: right;\"><strong>75.87<\/strong><\/td>\n<td style=\"text-align: right;\"><strong>72.11<\/strong><\/td>\n<td style=\"text-align: right;\"><strong>60.10<\/strong><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p>According to the publisher&#8217;s measurement results, despite having a relatively small pretraining dataset size of 346.03B source tokens, Cola2 scores higher in HellaSwag (41.75) and StoryCloze (80.42)\u2014which measure commonsense reasoning ability by choosing natural continuations of sentences\u2014than Llama-3-8B (39.21 and 79.49, respectively), which was trained on more than 43 times the amount of data.<\/p>\n<p>On the other hand, the average score across all 8 tasks (60.10) is the lowest among all compared models. In particular, in items such as MMLU (51.32) which evaluates broad and shallow general knowledge, and TruthfulQA (54.10) and ARC-Easy (87.62) which measure factuality such as resistance to misconceptions, the results fall short of the Qwen series, LLaDA-8B-Base, and Llama-3-8B.<\/p>\n<p>Note that it is explained that evaluation results may vary for each individual run because probabilistic elements are included in the latent variable generation process.<\/p>\n<h2>Strengths and Use Cases<\/h2>\n<p>Cola2 is provided as a base model rather than an instruction-tuned chat assistant. Therefore, rather than being used directly as a response engine for general conversational applications, it is suited for use cases such as hierarchical language modeling using a continuous latent space, research and experimentation on text generation algorithms based on Diffusion Transformer (DiT), and feature extraction.<\/p>\n<p>According to the model card description, despite a relatively small pretraining dataset size, it shows efficient results in commonsense reasoning tasks such as HellaSwag and StoryCloze, making it suitable for environments to verify the co-adaptation mechanism of continuous semantic compression and discrete generation, and to evaluate the performance of new generation architectures.<\/p>\n<p><!-- lmw:hardware --><\/p>\n<h2>Hardware Requirements<\/h2>\n<p><strong>Estimated requirements (calculated by Local Model Watch)<\/strong> \u2014 8.8B parameters<\/p>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Your VRAM<\/th>\n<th>Quantization<\/th>\n<th>File size<\/th>\n<th>Est. memory needed<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>48GB (RTX 6000 Ada \/ A6000, etc.)<\/td>\n<td>F32<\/td>\n<td>32.7GB<\/td>\n<td>39.2GB<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p><strong>Inference engine support<\/strong> (architecture name matched against each project&#8217;s own model registry in its source code, checked 2026-11-17): llama.cpp: not registered, vLLM: not registered, MLX (mlx-lm): not registered. &#8220;Not registered&#8221; means the name is absent from that registry today, not that the model cannot run.<\/p>\n<p><em>Memory estimates add a 20% runtime overhead (KV cache, etc.) to the actual size of the distributed files. Actual usage varies with context length, batch size and inference engine. These figures are computed by this site from file sizes, not published by the model&#8217;s authors. The listed files are distributed in F32 (32-bit). Loading in a lower precision (BF16 or quantized) needs less memory, so treat this as the size of the files as distributed, not a minimum. This site has not measured the lower-precision figures. Compare with other models in our <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/vram-guide-en\/\">VRAM quick reference<\/a>. What the quantization names mean: <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/glossary-quantization-en\/\">glossary<\/a>.<\/em><\/p>\n<p><!-- \/lmw:hardware --><\/p>\n<p><!-- lmw:runnability --><\/p>\n<h2>Can You Run It Locally?<\/h2>\n<p><strong>Not usable in Ollama, LM Studio and llama.cpp yet.<\/strong><\/p>\n<p>The publisher ships safetensors only, and llama.cpp&#8217;s registry does not list this architecture. llama.cpp would need to add support before these tools can run it. Today it can be run with transformers, using the memory figures in the table above.<\/p>\n<p><em>Compiled by this site&#8217;s code from the published formats, converted builds we have found, and each engine&#8217;s own model registry. &#8220;Not found&#8221; means we have not seen such a build, not that none exists.<\/em><\/p>\n<p><!-- \/lmw:runnability --><\/p>\n<p><!-- lmw:lab --><\/p>\n<h2>Our Own Measurements<\/h2>\n<h3>Measurements We Did Not Take<\/h3>\n<p>This model does not state a license, so we cannot confirm permission to use it. Because this site carries advertising, we did not run the model (no CPU run, answers, quantization comparison, conversion or generation).<\/p>\n<p><!-- \/lmw:lab --><\/p>\n<p><!-- lmw:peers --><\/p>\n<h2>Recent Models in the Same Size Class<\/h2>\n<p><em>Models with <\/em><em>4\u201315B<\/em><em> parameters that Local Model Watch covered recently, listed by code from our article log for comparison. VRAM tiers are this site&#8217;s estimates; licenses are as stated on the model cards.<\/em><\/p>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<p class=\"lmw-table-hint\" style=\"margin:0 0 4px;font-size:0.85em;opacity:0.7;\">\u2192 Scroll horizontally to see all columns<\/p>\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Model<\/th>\n<th>Parameters<\/th>\n<th>Smallest VRAM tier<\/th>\n<th>License<\/th>\n<th>Our article<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>microsoft\/UserRepGen-4B<\/td>\n<td>4.5B<\/td>\n<td>4GB<\/td>\n<td>mit<\/td>\n<td><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/userrepgen-4b\/\">UserRepGen-4B Text Generation Model: 4GB+ VRAM, GGUF Builds<\/a> (2026-11-17)<\/td>\n<\/tr>\n<tr>\n<td>IFM\/Jais-3-7B-GGUF<\/td>\n<td>9.0B<\/td>\n<td>8GB<\/td>\n<td>apache-2.0<\/td>\n<td><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/jais-3-7b-gguf-local-inference\/\">Jais-3-7B-GGUF Arabic and English 7B Reasoning Model: 8GB+ VRAM<\/a> (2026-11-17)<\/td>\n<\/tr>\n<tr>\n<td>IFM\/Jais-3-3.7B-GGUF<\/td>\n<td>5.1B<\/td>\n<td>4GB<\/td>\n<td>apache-2.0<\/td>\n<td><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/ifm-jais-3-37b-gguf\/\">Jais-3-3.7B-GGUF 512K Context Language Model: 4GB+ VRAM<\/a> (2026-11-17)<\/td>\n<\/tr>\n<tr>\n<td>ibm-granite\/granite-switch-4.1-8b<\/td>\n<td>8.8B<\/td>\n<td>24GB<\/td>\n<td>apache-2.0<\/td>\n<td><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/ibm-granite-switch-4-1-2\/\">IBM Releases Granite Switch 4.1: Modular LLMs with aLoRA and Speech<\/a> (2026-11-17)<\/td>\n<\/tr>\n<tr>\n<td>ggml-org\/NVIDIA-Nemotron-3.5-Bolt-E4B-GGUF<\/td>\n<td>8.0B<\/td>\n<td>12GB<\/td>\n<td>apache-2.0<\/td>\n<td><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/nvidia-nemotron-3-5-bolt-e4b-gguf-2\/\">NVIDIA-Nemotron-3.5-Bolt-E4B-GGUF Text Generation Model: 12GB+ VRAM<\/a> (2026-11-17)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p><!-- \/lmw:peers --><\/p>\n<h2>How to Get It<\/h2>\n<p>Cola2 model files are released in an 8-part safetensors format and can be obtained from Hugging Face. No special license agreement procedure (gated) is required for downloading.<\/p>\n<p>As execution environments, Python 3.10 or higher, PyTorch 2.2 or higher, Transformers 4.51 or higher and less than 5, safetensors 0.4 or higher, and the accelerate package are required, and a CUDA environment supporting bfloat16 is recommended. In addition, because this model has a unique structure, <code>trust_remote_code=True<\/code> must be specified when loading from Transformers.<\/p>\n<p>A minimal code example for loading the model from a Python script and performing inference is as follows:<\/p>\n<pre><code class=\"language-python\">import torch\nfrom transformers import AutoModel, AutoTokenizer\n\nmodel_id = &quot;ByteDance-Seed\/Cola2&quot;\ntokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)\nmodel = AutoModel.from_pretrained(\n    model_id,\n    trust_remote_code=True,\n    torch_dtype=torch.bfloat16,\n    low_cpu_mem_usage=True,\n).to(&quot;cuda&quot;).eval()\n<\/code><\/pre>\n<p>The following procedure can be used to download the complete set of model files from the command line and verify the file checksums:<\/p>\n<pre><code class=\"language-bash\">hf download ByteDance-Seed\/Cola2 --local-dir \/path\/to\/cola2\ncd \/path\/to\/cola2\nsha256sum -c SHA256SUMS\n<\/code><\/pre>\n<p>Note that command-line chat scripts, benchmark execution tools, and various documents are managed in the official <a href=\"https:\/\/github.com\/ByteDance-Seed\/Cola2\">GitHub repository<\/a>.<\/p>\n<p><!-- lmw:related --><\/p>\n<h2>Related Articles<\/h2>\n<ul>\n<li><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/bytedance-seed-qwen3-30b-a3b-moe-sae\/\">Qwen3-30B-A3B-MoE-SAE: 80GB+ VRAM<\/a><\/li>\n<\/ul>\n<p><!-- \/lmw:related --><\/p>\n<p><!-- lmw:next-steps --><\/p>\n<h2>What to Read Next<\/h2>\n<ul>\n<li><strong>Find models by VRAM<\/strong> (This model needs at least 48GB) \u2192 <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/vram-guide-en\/\">VRAM quick reference<\/a><\/li>\n<li><strong>Formats this model is available in<\/strong> \u2192 <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/format-safetensors-en\/\">Safetensors format guide and models<\/a><\/li>\n<li><strong>Learn about the publisher<\/strong> \u2192 <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/publisher-bytedance-en\/\">ByteDance Seed: models, licenses and articles<\/a><\/li>\n<li><strong>Other models for the same task<\/strong> \u2192 <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/models-by-task-en\/#task-text\">Other text generation models<\/a><\/li>\n<li><strong>How to read HellaSwag, TruthfulQA, MMLU<\/strong> \u2192 <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/glossary-benchmarks-en\/\">Benchmark glossary<\/a><\/li>\n<\/ul>\n<p><!-- \/lmw:next-steps --><\/p>\n<h2>Sources<\/h2>\n<ul>\n<li><a href=\"https:\/\/huggingface.co\/ByteDance-Seed\/Cola2\">ByteDance-Seed\/Cola2 &#8211; Hugging Face<\/a><\/li>\n<li><a href=\"https:\/\/arxiv.org\/abs\/2610.31305\">Cola2 Paper (arXiv:2610.31305)<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>ByteDance-Seed has released Cola2, a new language model combining semantic compression via a continuous latent space with discrete text generation.<\/p>\n","protected":false},"author":1,"featured_media":15300,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[310],"tags":[4636,4693,4695,4697,1833,4699,3668,1547,1555],"class_list":["post-15301","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-new-models","tag-bytedance-en","tag-bytedance-seed-en","tag-bytedance-seed-cola2-en","tag-cola2-en","tag-dit-en","tag-language-model-en","tag-vae-en","tag-verified","tag--en"],"lang":"en","translations":{"en":15301,"ja":15299},"_links":{"self":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts\/15301","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/comments?post=15301"}],"version-history":[{"count":2,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts\/15301\/revisions"}],"predecessor-version":[{"id":18064,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts\/15301\/revisions\/18064"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/media\/15300"}],"wp:attachment":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/media?parent=15301"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/categories?post=15301"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/tags?post=15301"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}