{"id":3533,"date":"2026-09-25T00:15:13","date_gmt":"2026-09-24T15:15:13","guid":{"rendered":"https:\/\/localmodelwatch.tsuchitsuchi.com\/2026\/09\/25\/nvidia-efficient-moe-training-biological-foundation-models\/"},"modified":"2026-09-25T18:07:19","modified_gmt":"2026-09-25T09:07:19","slug":"nvidia-efficient-moe-training-biological-foundation-models","status":"publish","type":"post","link":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/09\/25\/nvidia-efficient-moe-training-biological-foundation-models\/","title":{"rendered":"NVIDIA Optimizes MoE Training for Biological Foundation Models"},"content":{"rendered":"<p><!-- lmw:facts --><\/p>\n<h2>At a Glance<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Item<\/th>\n<th>Value<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Publisher<\/td>\n<td>NVIDIA Developer<\/td>\n<\/tr>\n<tr>\n<td>Published<\/td>\n<td>2026-09-25<\/td>\n<\/tr>\n<tr>\n<td>Source type<\/td>\n<td>Primary source (the publisher itself)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p><em>Values determined by this site&#8217;s code at collection time. Dates are JST.<\/em><\/p>\n<p><!-- \/lmw:facts --><\/p>\n<h2>Overview<\/h2>\n<p>NVIDIA has released an efficient Mixture-of-experts (MoE) training method for Biological Foundation Models. By leveraging the NVIDIA Transformer Engine (TE), reducing kernel launch overhead via GroupedLinear, suppressing memory usage with MXFP8, and employing kernel fusion techniques to integrate multiple operations, the training efficiency of MoE models is improved.<\/p>\n<h2>Claims and Evidence<\/h2>\n<p>NVIDIA claims that by using TE&#8217;s optimization primitives, challenges such as computation fragmentation, communication overhead, and memory usage in MoE architectures can be resolved. The specific technical approaches and evidence are as follows:<\/p>\n<ul>\n<li><strong>Computational efficiency via GroupedLinear<\/strong>: In the Hugging Face baseline implementation, overhead occurs because kernels are launched individually for each expert using Python loops. In contrast, TE&#8217;s <code>GroupedLinear<\/code> reduces launch and scheduling overhead by submitting GEMM (General Matrix Multiply) operations for multiple experts as a single grouped operation.<\/li>\n<li><strong>Memory reduction and hardware acceleration via MXFP8<\/strong>: Compared to BF16, MXFP8 can reduce memory usage because it represents weights and activations in 8 bits. MXFP8 employs block scaling, which assigns a scaling factor to blocks of 32 consecutive values, and is hardware-accelerated on NVIDIA Blackwell GPUs via dedicated Tensor Core instructions.<\/li>\n<li><strong>Overhead reduction via kernel fusion<\/strong>: Using TE&#8217;s <code>Sequential<\/code> API, <code>GroupedLinear<\/code>, <code>ScaledSwiGLU<\/code>, and routing weight scaling can be integrated into a single fused kernel called <code>ForwardGroupedMLP_CuTeGEMMSwiGLU_MXFP8<\/code>. This avoids the generation of intermediate data and reduces framework overhead associated with quantization and dequantization.<\/li>\n<\/ul>\n<p>In training benchmarks using BioNeMo recipes including these optimizations, the presenter reported that when using 8 NVIDIA B200 Tensor Core GPUs, a throughput up to 2.21 times higher was achieved compared to the Hugging Face baseline.<\/p>\n<h2>Specifications<\/h2>\n<p>The methods and results shown in this material include the following conditions:<\/p>\n<ul>\n<li><strong>Target Model<\/strong>: Mixtral-8x7B<\/li>\n<li><strong>Target Hardware<\/strong>: NVIDIA B200 Tensor Core GPU<\/li>\n<li><strong>Software\/Library<\/strong>: NVIDIA Transformer Engine (TE), PyTorch, BioNeMo recipe<\/li>\n<li><strong>Precision Setting<\/strong>: MXFP8 (When using NVIDIA Blackwell GPUs, the fused MXFP8 GroupedMLP kernel is available)<\/li>\n<li><strong>Parallelization Setting<\/strong>: Expert Parallelism (EP) is used. For 8 GPUs, a setting of EP=8 is shown.<\/li>\n<\/ul>\n<h2>How to Get It<\/h2>\n<p>Readers may be able to build and experiment with an MoE model training environment through the following resources and procedures:<\/p>\n<ul>\n<li><strong>Code and Recipes<\/strong>: It is recommended to try training MoE-based biological foundation models using the &#8220;Mixtral Native Transformer Engine recipe&#8221; included in NVIDIA BioNeMo Recipes.<\/li>\n<li><strong>Environment Setup<\/strong>: An NVIDIA CUDA-compatible environment is required. The environment can be built by using the provided Dockerfile or by installing the recipe requirements.<\/li>\n<li><strong>Execution Commands<\/strong>: The following commands are shown to verify the environment or execute training:<\/li>\n<\/ul>\n<p>Setting using 2 GPUs to confirm that expert parallelism and the training environment are working correctly:<\/p>\n<pre><code class=\"language-bash\">torchrun --nproc_per_node=2 train_fsdp2_ep.py --config-name L0_sanity\n<\/code><\/pre>\n<p>Setting for Mixtral-8x7B using 8 GPUs, applying expert parallelism (EP=8) and MXFP8 precision:<\/p>\n<pre><code class=\"language-bash\">torchrun --nproc_per_node=8 train_fsdp2_ep.py --config-name L1_8x7B_ep checkpoint.ckpt_dir=\/path\/to\/ckpt\n<\/code><\/pre>\n<p>It is explicitly stated that NVIDIA Blackwell GPUs are required to use the MXFP8 fused kernel.<\/p>\n<p><!-- lmw:related --><\/p>\n<h2>Related Articles<\/h2>\n<ul>\n<li><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/09\/24\/nvidia-releases-swe-serve-benchmark\/\">NVIDIA Releases SWE-Serve Benchmark for AI Coding Agents<\/a><\/li>\n<li><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/09\/23\/nvidia-blackwell-confidential-computing-ai-inference\/\">AI Inference Performance with NVIDIA Blackwell Confidential Computing<\/a><\/li>\n<li><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/09\/19\/nvidia-aiperf-benchmarking-llm-inference\/\">NVIDIA AIPerf: Benchmarking LLM Inference at Scale<\/a><\/li>\n<li><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/09\/15\/accelerating-dropless-moe-training-in-jax\/\">NVIDIA Accelerates Dropless MoE Training in JAX<\/a><\/li>\n<\/ul>\n<p><!-- \/lmw:related --><\/p>\n<p><!-- lmw:next-steps --><\/p>\n<h2>What to Read Next<\/h2>\n<ul>\n<li><strong>Models NVIDIA has released<\/strong> \u2192 <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/09\/10\/nvidia-releases-deepseek-v4-pro-nvfp4\/\">DeepSeek-V4-Pro-0813-nvfp4-DSpark: ~1005GB Memory<\/a><\/li>\n<\/ul>\n<p><!-- \/lmw:next-steps --><\/p>\n<h2>Sources<\/h2>\n<ul>\n<li><a href=\"https:\/\/developer.nvidia.com\/blog\/efficient-moe-training-for-biological-foundation-models\/\">Efficient MoE Training for Biological Foundation Models<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>NVIDIA uses Transformer Engine and MXFP8 to increase MoE training throughput by up to 2.21x on B200 GPUs for biological foundation models.<\/p>\n","protected":false},"author":1,"featured_media":3532,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1528],"tags":[933,2257,165,2259,695,1584,1209,1547],"class_list":["post-3533","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technical-reports","tag-bionemo-en","tag-mixtral-en","tag-moe-en","tag-mxfp8-en","tag-nvidia-en","tag-nvidia-developer-en","tag-transformer-engine-en","tag-verified"],"lang":"en","translations":{"en":3533,"ja":3531},"_links":{"self":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts\/3533","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/comments?post=3533"}],"version-history":[{"count":2,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts\/3533\/revisions"}],"predecessor-version":[{"id":4402,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts\/3533\/revisions\/4402"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/media\/3532"}],"wp:attachment":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/media?parent=3533"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/categories?post=3533"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/tags?post=3533"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}