Open-source AI infrastructure project llm-d announced version 1.0 on October 20. The release gives organizations a way to run large language models across their own infrastructure, cloud environments, and accelerator hardware while addressing the cost and operational challenges of serving AI models at scale.
Production support for AI models
llm-d is a Kubernetes-native system for distributed inference, the process of generating answers from trained AI models. IBM Research says contributors include IBM, Red Hat, Google, CoreWeave, NVIDIA, AMD, Cisco, Hugging Face, Intel, Lambda, and Mistral AI.
Version 1.0 focuses on the work required after an AI service launches. It adds a stable API, support for multiple users and workloads sharing infrastructure, and upgrades designed to avoid interrupting service. The project works with inference engines such as vLLM and SGLang.
Lower costs for busy workloads
To improve efficiency, the framework routes requests to servers that already hold related context, moves cached model data between GPU, CPU, and storage, separates prompt processing from response generation, and distributes large models across multiple accelerators.
IBM said a recent agent-based coding test achieved 97% reuse of available cache. IBM and Red Hat also reported serving 3,000 software-engineering agents on self-hosted infrastructure at five to 10 times less than the actual price of serving the same open model as a service. These are vendor-reported results, not independent benchmarks.
Broader options for enterprises
These capabilities give organizations an alternative to relying entirely on one model provider, cloud company, or hardware platform. The project's GitHub repository lists production routing, autoscaling, batch processing, support for modern accelerators, and an Apache 2.0 license.
The immediate audience is developers and infrastructure teams building large-scale AI services. The project could give companies more control over AI infrastructure spending and the systems used to run sensitive workloads.
References
Hess, P. (2026, October 20). llm-d 1.0 hardens open-source AI inference for day 2 production operations. IBM Research. https://research.ibm.com/blog/llmd-1-0-is-ready-for-production-inference-serving
llm-d. (n.d.). llm-d: Achieve state of the art inference performance with modern accelerators on Kubernetes [Source code]. GitHub. Retrieved October 20, 2026, from https://github.com/llm-d/llm-d