202607111246
Status: #idea
Tags:

Deploying Large Models at Scale with SGLang

• Miles → Post training framework

SGLang

• KV-cache offloading is native in SGLang
• Context parallelism
• Pool-per-stage deployments (vs PD-disaggregated)
  • MI300 is better at decode vs Hopper chips

Stack

• RadixAttention
  • TODO
• HiCache
  • KVCache offloading
  • Why does hi-cache need deterministic attention?
• HiSparse
• ShadowRadix

Deepseek V4

• DeepSeek V4 attention
  • Hybrid
  • TODO ask for presentation
• Kernels
  • Lightning TopK
    • Cluster of 8, 10-bit radix histogram replaces a global sort
  • MegaMoE → Dispatch, compute and combine into a single kernel

Miles

Harbor → For agentic rollouts
TODO from slide (Inside miles post training as scale)

References

1. https://www.linkedin.com/in/alex-nails/