Making models fast

Big models are expensive to run

A model with 70 billion parameters stored as 32-bit numbers needs 280 GB of memory just to hold its weights — and every generated token touches all of them. Serving millions of people makes efficiency as important as intelligence. Three tricks do much of the heavy lifting.