How We Make AI Inference Fast - Performance-Maxing GPUs to Their Physical Limits

202607111326
Status: #idea
Tags:

Power utilised can act as a better proxy for GPU utilisation that the one from nvidia-smi
Serving layer should not have interpreters on the hot path
Don’t look at mean/median. Look at the tail, in order to see performance relative to the
In speech, OCR, vision, there are multiple models
- Components
  - VLM = vision encoder + projector + LLM
  - Speech = AR generator + codec
  - ASR = encoder/decoder + rescorer
  - Diffusion = text encoder + denoiser + VAE
- Each sub-model has its own baseline.
- Batching domains differ
  - Some models need streaming input, some need batched inputs
Q (Rakesh): Why doesn’t CUDA graphs work for serving small models?
Kernel optimisation
- NCU Metrics of interest
  - SM
  - DRAM utilisation
  - Tensor-pipe utilisation
- Formulation
  - E.g. Use Taylor series expansion for $\sin x$ and run it on the tensor core
Q (Rakesh): How does the agent store the learnings?

References