<\/p>\n
| Item<\/th>\n | Value<\/th>\n<\/tr>\n<\/thead>\n | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Publisher<\/td>\n | NVIDIA Developer<\/td>\n<\/tr>\n | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Published<\/td>\n | 2026-09-24<\/td>\n<\/tr>\n | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Source type<\/td>\n | Primary source (the publisher itself)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n Values determined by this site’s code at collection time. Dates are JST.<\/em><\/p>\n <\/p>\n NVIDIA has announced SWE-Serve, a new benchmark designed for AI coding agents. Derived from pull requests in SGLang, an open-source LLM inference serving system, this benchmark uses 53 tasks to evaluate whether code modifications in inference engineering function correctly through actual serving paths (including model loading and request processing).<\/p>\n According to the presenters’ measurements, a “serving gap” has been demonstrated where patches created by AI agents may pass local tests yet fail in actual live serving environments.<\/p>\n In 19 of the tasks included in SWE-Serve, actual servers are launched to perform live serving checks. The presenters’ measurements showed that for these tasks, 69.4% of patches passed when live serving checks were excluded, whereas only 45.9% passed when a complete verifier was used. This indicates that about one-third of patches that passed other tests failed live serving validation. For example, in tasks related to Gemma 4 MoE, while passing all other checks, 16 out of 33 patches failed at least one live serving test, such as model loading, expert routing, text or image serving, or batch generation in the correct order.<\/p>\n Significant differences in pass rates were observed depending on the scope of execution domains covered by the tasks. For 27 tasks spanning multiple execution domains\u2014ranging from request processing and I\/O to scheduling, model execution, and KV cache management\u2014the pass rate was 47.7%, which is 21.3 percentage points lower compared to the 26 tasks limited to a single domain (69.0% pass rate). This trend is similarly confirmed across all tested models.<\/p>\n In evaluations using mini-swe-agent, average pass@1 scores across 11 models ranged from 34.6% to 75.5%. Details of the settings where each model recorded its highest score are as follows:<\/p>\n \u2192 Scroll horizontally to see all columns<\/p>\n Additionally, it was shown that even among models with equivalent performance, the average cost per task and execution time vary significantly.<\/p>\n The evaluation results for this benchmark are based on the following conditions:<\/p>\n Readers can evaluate their own coding agents on SGLang inference engineering tasks through the following resources:<\/p>\n <\/p>\n <\/p>\n |