What is a TPU?

A TPU, or tensor processing unit, is a chip Google designed to make machine learning faster. Where a general-purpose processor tries to be good at everything, a TPU is built for one kind of work: the large matrix calculations at the heart of neural networks.

How it works

Training and running an AI model mostly means multiplying big grids of numbers together, over and over. TPUs are application-specific chips (ASICs) built around that job:

• A matrix unit: each TPU core has a matrix multiply unit arranged as a 128×128 grid (a "systolic array") that streams numbers through and multiplies them in parallel.
• Fast on-chip memory: high-bandwidth memory (HBM) sits right next to the compute, so larger models and batches fit without slowing down.

TPU vs. CPU vs. GPU

• CPUs are flexible and good for quick prototyping or models with lots of custom operations.
• GPUs are strong general-purpose accelerators and handle custom operations well.
• TPUs are at their best on long, large-scale training and serving jobs dominated by matrix math.

How people use them

You don't buy a TPU off the shelf. Google offers them in its cloud, through services such as Compute Engine and Google Kubernetes Engine. Google has released several generations, including v2, v3, v4, v5e, v5p, v6e and the newer Ironwood (TPU7x). A recent generation, Trillium, is also the chip Google is testing in orbit with Project Suncatcher.