AI Engineer · AI

Weight Folding, CUDA Streams, and the Bug That Made My Model Speak Backwards — Filip Makraduli

An RMS norm layer does almost none of the arithmetic in a transformer, yet a single decode step can launch it around 33 times, and a GPU is fast at math and slow at everything else: starting work, moving data, waiting. That gap is what FlashNorm attacks. Filip Makraduli wrote the paper with Nils Graef, and the idea fits in two lines of algebra. Fold the norm's gain into the projection weights offline so one matrix absorbs both. Defer the scalar d

Introductie van de bron.