Split Models Across Nodes When They Exceed Single-machine Capacity

Rust-based distributed LLM inference engine that pools compute across machines and exposes an OpenAI-compatible API. It routes requests across peers and can split large models into stages for multi-node serving.

AI Frameworks | Rust · distributed inference · LLM serving