{"id":14650,"date":"2026-11-17T11:03:12","date_gmt":"2026-11-17T02:03:12","guid":{"rendered":"https:\/\/localmodelwatch.tsuchitsuchi.com\/?p=14650"},"modified":"2026-11-17T11:12:37","modified_gmt":"2026-11-17T02:12:37","slug":"eleutherai-pythia-160m-weight-seed2-corrected-2","status":"publish","type":"post","link":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/eleutherai-pythia-160m-weight-seed2-corrected-2\/","title":{"rendered":"EleutherAI Releases Pythia 160M Weight Seed 2 Corrected"},"content":{"rendered":"<p><!-- lmw:facts --><\/p>\n<h2>At a Glance<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Item<\/th>\n<th>Value<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Repository<\/td>\n<td><a href=\"https:\/\/huggingface.co\/EleutherAI\/pythia-160m-weight-seed2-corrected\">EleutherAI\/pythia-160m-weight-seed2-corrected<\/a><\/td>\n<\/tr>\n<tr>\n<td>Publisher guide<\/td>\n<td><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/publisher-eleutherai-en\/\">EleutherAI: models and licenses<\/a><\/td>\n<\/tr>\n<tr>\n<td>Published<\/td>\n<td>2026-11-02<\/td>\n<\/tr>\n<tr>\n<td>License<\/td>\n<td>apache-2.0<\/td>\n<\/tr>\n<tr>\n<td>Formats<\/td>\n<td>PyTorch (bin)<\/td>\n<\/tr>\n<tr>\n<td>Source type<\/td>\n<td>Primary source (the publisher itself)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p><em>Values determined by this site&#8217;s code when the information was collected. Dates are JST.<\/em><\/p>\n<p><!-- \/lmw:facts --><\/p>\n<h2>Overview<\/h2>\n<p>EleutherAI has released <code>EleutherAI\/pythia-160m-weight-seed2-corrected<\/code>, a corrected version of the Pythia series model.<\/p>\n<p>This model is newly trained from an independent weight initialization using seed-2, while keeping the training data presentation order fixed from the original Pythia.<\/p>\n<p>Previously released weight-seed versions of the model contained a training bug. Because they continued processing from the original Pythia weights while retaining inaccurate Adam optimizer buffers, the initialization changes were not actually applied. This release fixes that training error, correctly providing the experimental environment intended by the independent initialization.<\/p>\n<p>Training was performed using The Pile dataset over 143,000 steps with a global batch size of 1,024 sequences and a context length of 2,048 tokens. In addition, all 154 checkpoints from step 0 to step 143,000 have been published as step revisions, and this model revision includes the final step 143000 checkpoint.<\/p>\n<h2>Specifications<\/h2>\n<ul>\n<li>Parameters: 160M<\/li>\n<li>Architecture: GPTNeoXForCausalLM (<code>gpt_neox<\/code>)<\/li>\n<li>Context length: 2,048 tokens<\/li>\n<\/ul>\n<h2>Strengths and Use Cases<\/h2>\n<p>This model is primarily intended for research purposes to evaluate and verify the effects of weight initialization (random seeds) and training reproducibility in language model pre-training.<\/p>\n<p>As described in the model card and indicated by the tags (<code>reproducibility<\/code>, <code>en<\/code>, etc.), it is not a general-purpose language model aimed at advanced task processing in practical work or services, but rather an experimental model designed to elucidate LLM behavior and analyze the learning process. It is trained by initializing weights with an independent seed-2 setting while maintaining the exact same training data presentation order (The Pile) as the original Pythia model. This allows for a proper comparative verification of how differences solely in the initialization state affect model performance and acquired representations, while keeping the training data order conditions constant.<\/p>\n<p>In particular, this release is published as a fix for the training errors that occurred in past weight-seed versions. In previous versions, learning continued from the original Pythia weights with the Adam optimizer buffers incorrectly remaining, so the initialization change was not substantially implemented. With this release, the originally intended independent initialization experimental environment is correctly reproduced, enabling accurate data collection and analysis regarding the effects of initialization.<\/p>\n<p>Furthermore, not only the final step 143000 model itself, but all 154 checkpoints from the start of training at step 0 up to step 143,000 are saved and provided as step revisions. This makes it suitable for experiments that track in detail the learning dynamics of how model parameters and representations transition and change at each step of training progression, rather than simply evaluating final performance after training completion.<\/p>\n<p>Note that the parameter count is a small model configuration of 160M (160 million), and it is not a model expected to have advanced practical performance in complex reasoning, code generation, or dialogue tasks. Its use is strictly intended for reproducibility research and behavioral analysis of learning algorithms.<\/p>\n<h2>How to Get It<\/h2>\n<p><code>EleutherAI\/pythia-160m-weight-seed2-corrected<\/code> is published and distributed on the Hugging Face Hub.<\/p>\n<p>The license is set to Apache 2.0 (<code>apache-2.0<\/code>). There are no access restrictions (gated) requiring agreement procedures for terms of use or prior applications, and anyone can directly access and download the model data.<\/p>\n<p>The model distribution format supports the PyTorch format (<code>PyTorch (bin)<\/code>), and it can be loaded using standard frameworks such as Hugging Face&#8217;s <code>transformers<\/code> library.<\/p>\n<p>In addition to the standard final step 143000 checkpoint provided, by specifying the revision feature during download, all 154 checkpoints from step 0 to step 143,000 can be individually selected and acquired.<\/p>\n<p><!-- lmw:runnability --><\/p>\n<h2>Can You Run It Locally?<\/h2>\n<p><strong>Not usable in Ollama, LM Studio and llama.cpp yet \u2014 we have found no GGUF build.<\/strong><\/p>\n<p>The publisher ships PyTorch (bin) only. However, llama.cpp&#8217;s registry does list this architecture, so <strong>conversion to GGUF is possible<\/strong> and the model will run once someone publishes a converted build.<\/p>\n<p><strong>License \u2014 <code>apache-2.0<\/code> (Commercial use allowed):<\/strong> Permits commercial use, modification and redistribution. Redistribution requires including the license and stating changes; includes a patent grant.<\/p>\n<p><em>Compiled by this site&#8217;s code from the published formats, converted builds we have found, and each engine&#8217;s own model registry. &#8220;Not found&#8221; means we have not seen such a build, not that none exists. License summaries are not legal advice \u2014 check the publisher&#8217;s original terms before relying on them.<\/em><\/p>\n<p><!-- \/lmw:runnability --><\/p>\n<p><!-- lmw:related --><\/p>\n<h2>Related Articles<\/h2>\n<ul>\n<li><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/09\/16\/eleutherai-olmo-3-7b-reward-hacking-models\/\">EleutherAI Releases OLMo-3-7B Models for Reward-Hacking Research<\/a><\/li>\n<li><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/09\/15\/eleutherai-releases-bergson-leaderboard-baseline-gpt2-model\/\">EleutherAI Releases Bergson Leaderboard Baseline GPT-2 Model<\/a><\/li>\n<\/ul>\n<p><!-- \/lmw:related --><\/p>\n<p><!-- lmw:next-steps --><\/p>\n<h2>What to Read Next<\/h2>\n<ul>\n<li><strong>Engines that run this model<\/strong> \u2192 <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/engine-vllm-en\/\">vLLM<\/a><\/li>\n<li><strong>Learn about the publisher<\/strong> \u2192 <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/publisher-eleutherai-en\/\">EleutherAI: models, licenses and articles<\/a><\/li>\n<li><strong>Other models for the same task<\/strong> \u2192 <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/models-by-task-en\/#task-text\">Other text generation models<\/a><\/li>\n<\/ul>\n<p><!-- \/lmw:next-steps --><\/p>\n<h2>Sources<\/h2>\n<ul>\n<li><a href=\"https:\/\/huggingface.co\/EleutherAI\/pythia-160m-weight-seed2-corrected\">https:\/\/huggingface.co\/EleutherAI\/pythia-160m-weight-seed2-corrected<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>EleutherAI has released the corrected weight-seed 2 version of the Pythia 160M model for reproducibility research.<\/p>\n","protected":false},"author":1,"featured_media":14649,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[310],"tags":[373,4507,4509,4511,1547,1555],"class_list":["post-14650","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-new-models","tag-eleutherai-en","tag-pythia-en","tag-pythia-160m-weight-seed2-corrected-en","tag-pythia-weight-seed2-corrected-en","tag-verified","tag--en"],"lang":"en","translations":{"en":14650,"ja":14648},"_links":{"self":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts\/14650","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/comments?post=14650"}],"version-history":[{"count":2,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts\/14650\/revisions"}],"predecessor-version":[{"id":18020,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts\/14650\/revisions\/18020"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/media\/14649"}],"wp:attachment":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/media?parent=14650"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/categories?post=14650"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/tags?post=14650"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}