Mooncake: Shared KV Cache and Data Movement for Distributed AI Inference and RL

Distributed AI workloads need to move and reuse large amounts of data across inference and training workers. This session introduces Mooncake Store and Transfer Engine, covering shared KV cache storage, prefill/decode disaggregation, and high-performance data transfer. We will discuss integration with vLLM and SGLang, Kubernetes deployment through RBG, and recent progress in RL rollout data movement and heterogeneous weight updates. The discussion will focus on architecture, practical deployment considerations, and opportunities for collaboration across the cloud-native AI ecosystem.

Outline (30 minutes, followed by Q&A):

• 5 min: data movement and cache reuse in distributed AI workloads
• 10 min: Mooncake Store, Transfer Engine, and inference integrations
• 10 min: recent RL and heterogeneous weight-update progress
• 5 min: Kubernetes deployment considerations and collaboration opportunities

Speaker: Teng Ma, Mooncake maintainer