What is AI inference?

AI inference is the moment an AI model does its job. The model has already been trained; inference is when it takes something new, like your question, a photo or a sensor reading, and produces an answer. Every chatbot reply, image recognition and recommendation is an inference.

Training vs. inference

• Training is learning. Data flows through the network forward and backward while the model adjusts its internal weights, over and over. It is enormously expensive: NVIDIA says training Meta's Llama 3.1 405B took about 3.8 × 10²⁵ math operations.
• Inference is using what was learned. Data flows only forward, and the goal is to answer quickly and cheaply.

For a chatbot, inference has two steps: the model reads your whole prompt at once (called prefill), then writes the reply one token at a time (called decode).

Why it matters now

Training happens once per model. Inference happens every time anyone uses it, so as AI apps grow, serving answers becomes a huge share of the work, and NVIDIA says inference performance directly affects AI providers' profits.

Reasoning models push this further. They "think" before answering by spending extra compute at answer time, which NVIDIA calls test-time scaling. On a hard question that can take more than 100 times the compute of a single normal answer.

Where it runs

• Cloud data centers, on GPUs, Google's TPUs and custom chips. AWS, for example, builds its own Inferentia chips just for inference.
• On-device and at the edge, on phones, laptops and cars with built-in AI chips. That's faster and more private, but limited by the hardware.

Speed matters as much as cost: chatbots and self-driving cars need answers in real time, while jobs that can wait are often run in cheaper batches.