EleutherAI Releases Bergson Leaderboard Baseline GPT-2 Model
At a Glance
| Item | Value |
|---|---|
| Repository | EleutherAI/bergson-wikitext-gpt2-leaderboard |
| Published | 2026-09-15 |
| License | apache-2.0 |
| Formats | safetensors |
| Paper | arXiv:2303.14186 |
| Source type | Primary source (the publisher itself) |
Values determined by this site’s code at collection time. Dates are JST.
Overview
EleutherAI has released “EleutherAI/bergson-wikitext-gpt2-leaderboard", the baseline model for the training data attribution benchmark “bergson leaderboard".
This model is based on openai-community/gpt2 and has been fine-tuned for 4 epochs using 4,608 chunks of the WikiText dataset (EleutherAI/bergson-wikitext-512-chunks). According to the model card, the held-out loss has decreased from 3.545 to 3.111.
Specifications
- Base model: openai-community/gpt2
- License: apache-2.0
Performance
Publishing model card details, scores from the bergson leaderboard evaluating various data attribution methods are disclosed. The table provided is as follows:
| Method | Proponent QLD [95% CI] | LDS [95% CI] |
|---|---|---|
| MAGIC | 0.100 [0.090, 0.112] | 0.931 [0.925, 0.936] |
| Eigenvalue-corrected Shampoo | 0.071 [0.060, 0.082] | 0.517 [0.491, 0.539] |
| EK-FAC | 0.070 [0.058, 0.082] | 0.454 [0.426, 0.479] |
| KFAC | 0.067 [0.056, 0.080] | 0.420 [0.391, 0.446] |
| BM25 | 0.062 [0.048, 0.076] | 0.220 [0.185, 0.252] |
| Qwen3-Embedding-8B semantic search | 0.049 [0.038, 0.061] | 0.132 [0.093, 0.169] |
| TrackStar (no optimizer correction, projection 64) | 0.045 [0.036, 0.055] | 0.270 [0.240, 0.295] |
| TRAK (8-model ensemble) | 0.032 [0.024, 0.040] | 0.138 [0.111, 0.165] |
| SOURCE (Adam) | 0.024 [0.018, 0.030] | 0.154 [0.126, 0.181] |
| Gradient cosine similarity | 0.021 [0.016, 0.027] | 0.156 [0.131, 0.181] |
| Activation similarity | 0.000 [-0.000, 0.001] | 0.110 [0.070, 0.149] |
According to the explanations of the metrics provided by the publishers, LDS (Linear datamodeling score) indicates the accuracy of global data ranking methods based on influence, while QLD (Query loss difference) shows how much the model loss increases for held-out queries when top 1% influence data is removed and retrained compared to a random exclusion baseline.
Based on the scores in the table, MAGIC recorded 0.100 [0.090, 0.112] in Proponent QLD and 0.931 [0.925, 0.936] in LDS, representing the highest values among all methods. On the other hand, Activation similarity yielded the lowest results with 0.000 [-0.000, 0.001] for Proponent QLD and 0.110 [0.070, 0.149] for LDS. Additionally, the Proponent QLD scores for EK-FAC and Eigenvalue-corrected Shampoo were 0.070 and 0.071 respectively, showing a very close margin.
Regarding the file structure included in the repository, the model card provides the following correspondence table:
| path | what it is |
|---|---|
exported/checkpoint-72/ |
the final model, scored by every leaderboard method |
exported/checkpoint-{12,24,36,48,60}/ |
interval checkpoints with optimizer.pt, used by SOURCE and the checkpoint-averaged variants |
optimizer.pt, config.yaml |
final AdamW second moments (TrackStar-Adam) and the training config |
trak_ensemble/s{0..3}_subset_{0,1}/ |
the eight GPT-2 models trained on independent random 50% subsets for the TRAK row (trak_ensemble/train_s*.yaml) |
Strengths and Use Cases
According to the model card and metadata, this model is intended for use cases related to training-data-attribution, influence-functions, and model interpretability.
Rather than serving as a general-purpose conversational or coding model for specific tasks, it is designed as a benchmark model to measure and compare the impact that various data attribution methods have on the model within the bergson leaderboard. The retraining bank and set of score results are provided at EleutherAI/bergson-wikitext-gpt2-leaderboard-bank.
Hardware Requirements
Estimated requirements (calculated by Local Model Watch)
| Your VRAM | Quantization | File size | Est. memory needed |
|---|---|---|---|
| 8GB (RTX 4060 / 3060 Ti, etc.) | そのままの精度 | 6.5GB | 7.8GB |
Memory estimates add a 20% runtime overhead (KV cache, etc.) to the actual size of the distributed files. Actual usage varies with context length, batch size and inference engine. These figures are computed by this site from file sizes, not published by the model’s authors. Compare with other models in our VRAM quick reference. What the quantization names mean: glossary.
How to Get It
This model is available for download from Hugging Face. The distribution format includes safetensors, and since the repository is not gated, it can be acquired without any prior application or agreement procedures.
Related Articles
Sources
- https://huggingface.co/EleutherAI/bergson-wikitext-gpt2-leaderboard
- https://bergson.readthedocs.io/en/latest/leaderboard.html
- https://huggingface.co/datasets/EleutherAI/bergson-wikitext-512-chunks
- https://github.com/EleutherAI/bergson/tree/main/examples/compare_wikitext/2_interval.yaml
- https://github.com/EleutherAI/bergson
- https://huggingface.co/datasets/EleutherAI/bergson-wikitext-gpt2-leaderboard-bank
- https://arxiv.org/abs/2303.14186
- https://github.com/EleutherAI/bergson/tree/main/examples/compare_wikitext
- https://huggingface.co/collections/EleutherAI/data-attribution-6a8d013ccc372b7fd6abce3e
- https://huggingface.co/spaces/mteb/leaderboard
- https://github.com/EleutherAI/bergson/tree/main/examples/compare_wikitext/trak_ensemble