AI Engineer · AI

Operating Distributed Inference Systems at Scale — Nishant Gupta & Naman Ahuja, Meta

Inference traffic at Meta already outpaces the largest microservices in the world, the fastest growing workload it has ever run. Nishant Gupta's frame for what comes next is 2008. The cloud began as virtual machines, then value climbed to schedulers, service meshes and autoscalers, until orchestration held the complexity and the value. AI is on the same path, compressed into a few years, with orchestration emerging around routing, KV cache manage

Introductie van de bron.