Pool Gpus And Memory Across Machines To Serve Llms

Rust-based distributed LLM inference engine that pools compute across machines and exposes an OpenAI-compatible API. It routes requests across peers and can split large models into stages for multi-node serving.

AI Frameworks | Rust · distributed inference · LLM serving