IEEE Spectrum breed · Technology

AI Inference Takes Center Stage, Driving Hardware Innovation

While AI training dominated recent years, the focus has shifted to inference – using trained models to generate outputs. This surge in demand is reshaping the AI hardware landscape.

AI models like GPT-4o have dramatically improved performance, with GPT-4o scoring 88.7% on a knowledge benchmark, up from GPT-3's 43.9% in 2020. This progress has made LLMs more useful, leading to widespread adoption and a shift in industry focus from training to inference.

Inference demand is escalating due to reasoning models and agentic AI. This has spurred collaborations, such as OpenAI and Amazon using Cerebras chips, and Nvidia acquiring talent from Groq.

AI inference differs computationally from training. Training organizes parameters; inference uses them to generate outputs. Inference faces challenges due to the autoregressive nature of models, where each new token depends on previous ones.

Inference has two phases: prefill and decode. Prefill is parallelizable. Decode, generating tokens sequentially, is slower as it requires rereading model weights from memory for each token.

This memory bottleneck drives innovation. Companies like d-Matrix stack compute and memory, while Majestic Labs improves memory interfaces. Both explore DRAM to cut costs compared to HBM.

Major players adopt multi-chip strategies. Nvidia combines GPUs with LPUs. AWS pairs accelerators with Cerebras's engine. These systems pool different chips for inference.

Software optimizations like quantization are crucial. New 4-bit formats improve performance with minimal quality loss. Startups explore novel approaches like logarithmic number systems and specialized silicon.

AI-samenvatting op basis van de bron.