What is Inference?
Also called model serving, prediction.
Inference is the act of running a trained model to produce output, as distinct from training, which creates the model. During inference the parameters are frozen and no learning occurs. Every request sent to a deployed model is an inference call, and inference is where the ongoing operational cost of running a model accumulates over its lifetime.
Text generation is autoregressive, meaning the model produces one token, appends it to the sequence, and runs again to produce the next. A response of five hundred tokens therefore requires five hundred sequential passes through the network. This is why longer answers take proportionally longer to arrive and why output tokens are usually metered at a higher rate than input tokens.
Serving splits into two phases with different characteristics. Prefill reads the entire prompt in parallel and is limited by compute, while decoding emits tokens one at a time and is limited by memory bandwidth. Optimizations such as caching intermediate values, batching many users into one pass, and speculative decoding target these two phases separately rather than uniformly.
A frequent misconception is that a model learns from the conversations it serves. Standard inference changes nothing inside the model. Any apparent learning comes from information placed back into the prompt, from an external retrieval store, or from a separate training run performed later on collected data, subject to whatever data retention policy the provider and customer agreed on.
Because inference recurs with every request while training is paid once, total spend on inference typically exceeds training cost over a deployed system's life. That economics drives most practical model choices: routing simple requests to smaller models, caching repeated prompts, quantizing weights, and capping output length are all fundamentally inference cost controls rather than quality decisions.
Key points
- Running a trained model; parameters stay frozen and nothing is learned.
- Text is generated one token at a time, sequentially.
- Prefill reads the prompt in parallel; decoding emits tokens serially.
- Ongoing inference spend usually outweighs one-time training cost.
In practice
You ask a deployed model to summarize a report. The service reads your entire prompt in one parallel pass, then begins emitting the summary token by token, each step conditioned on everything written so far. After roughly two hundred steps it emits a stop signal and the response ends. The model's weights are byte for byte identical before and after your request.