What this talk is about
Cocktail is an ensemble-based model-serving framework for the public cloud that simultaneously optimises three things production teams actually care about:
- Cost — by leveraging transient cloud instances and dynamically selecting the smallest viable subset of models per request.
- Accuracy — by combining predictions across the active ensemble with a weighted majority vote.
- Latency / SLO — by predicting per-request resource needs and provisioning ahead of demand spikes.
The talk walks through the per-component optimisations and the joint control loop that ties them together. It also covers the decoupling between model-selection (a slow loop) and request-routing (a fast loop) that makes the system responsive without flapping.
Quick references
- Paper: Publications page → Cocktail (NSDI ‘22)
- Slides: TODO add a public link if you want them embedded inline.