Kv-cache-aware Prefix Routing To Maximize Gpu Utilization

SMG is a Rust LLM gateway that unifies OpenAI/Anthropic/Gemini and self-hosted engines (vLLM, SGLang, TensorRT-LLM, TokenSpeed, MLX) behind one API. It adds KV-cache-aware routing, gRPC pipelines, MCP tooling, multi-tenancy and observability for large-scale inference deployments.

AI Frameworks | Rust · llm-gateway · inference-routing