GLM Announces Custom Inference Infrastructure and Local MoE Tests
Overview
GLM (z.ai) has announced the construction of its own proprietary inference infrastructure. According to the documentation, a commercial-grade inference service has been built from scratch on a cluster consisting of over 100,000 Chinese-made AI accelerators. It is reported that all production inference for the “GLM-5.3-Flash" model runs on this infrastructure.
Announcement Details
According to the official blog post “GLM Built Its Own Inference Infrastructure" and excerpts shared within the community, the announcement centers on the operation of this newly developed, large-scale inference foundation.
The system is built from the ground up on a cluster comprising over 100,000 Chinese-made AI accelerators. All production inference processing for “GLM-5.3-Flash" is executed on this system. The documentation explains that the system implementation incorporates a series of aggressive memory optimizations.
On the other hand, regarding the user experience via the official z.ai service, the community has pointed out slow inference speeds and strict usage limits. Mentions were also made regarding pricing changes, such as the cheapest plan being set at around $20 per month and the mid-tier plan at around $80 per month.
Background
Detailed background information on why the proprietary inference infrastructure was built is not directly stated in the materials beyond the context of the official blog. However, community discussions point to a connection with the current situation where Chinese companies are rapidly advancing software optimization for domestic AI chips and infrastructure, driven by US semiconductor export controls and related factors. Discussions also noted that, compared to previous GLM announcements, this release focuses on the technical aspects of software and infrastructure optimization.
Impact on Local LLM Users
While this announcement concerns GLM’s commercial cloud inference infrastructure, it provides several pieces of information relevant to engineers running LLMs in local environments.
First, while the official inference infrastructure was optimized for custom accelerators, the documentation contains no specific changes or mentions regarding future policies for releasing open-weight models or the distribution formats and license conditions of existing models.
Meanwhile, as an approach to running models of the same lineage in self-hosted environments, a noteworthy implementation example has been reported by the community. For “GLM-5.3" (744B MoE, 4-bit experts, 434 GB on disk), validation results were shared using “argodrive" (built on antirez/ds4), a tool that streams expert weights from an NVMe SSD for inference.
According to this report, in a single MacBook Pro M5 Max environment with 128 GB of memory, running a 744B-scale model is made possible by streaming experts from the NVMe SSD instead of keeping them resident in main memory. The recorded inference speeds are as follows:
- Single NVMe drive configuration: approx. 2 tok/s
- Configured with striping across 4 NVMe drives: 3.5 tok/s (maintaining byte-for-byte identical output)
- Internal build with unpublished patches applied: 4.2 tok/s
Thus, while cloud-side inference optimization is progressing via large-scale clusters, attempts are also being driven by the engineering community to run large MoE models locally using a combination of limited memory and high-speed storage.