A GPU (graphics processing unit) is a general-purpose parallel processor built by companies like NVIDIA and AMD, while a TPU (tensor processing unit) is Google’s custom chip designed specifically to accelerate the matrix multiplications that power neural networks. For most teams, a GPU is the safer default because of its broader software support, multi-cloud availability, and the depth of the CUDA ecosystem. A TPU beats a GPU on large-scale training of transformer models inside Google Cloud, particularly when your code runs on JAX or TensorFlow and you can afford the migration cost.
What a GPU Actually Does in an AI Workload
A GPU started life as a chip for rendering polygons. What made it useful for AI is the same thing that made it useful for graphics: thousands of small, parallel compute cores that can perform many floating-point operations simultaneously. NVIDIA’s data-center-class chips added Tensor Cores starting with the Volta architecture in 2017, which accelerated the specific matrix-multiply-accumulate operations that dominate neural network training and inference.
The real competitive advantage of the GPU in AI is not raw compute. It is the software stack built around it. CUDA, NVIDIA’s proprietary parallel computing platform, has been in production since 2007 and has accumulated an enormous library of optimized kernels, profiling tools, and framework integrations. PyTorch and most of the broader open-source ecosystem are built against CUDA first. When a new model architecture emerges, a CUDA kernel for it appears within days. That density of tooling is genuinely hard to replicate.
You can run GPUs on every major cloud, AWS, Azure, Google Cloud, Oracle, CoreWeave, Lambda Labs, and on bare metal in your own data center. Comparing NVIDIA’s H100 and H200 data center GPUs shows how much performance scales across generations even within a single vendor’s lineup. That portability matters enormously when your cloud spend, latency requirements, or data residency rules shift.
What a TPU Is and How Google Builds Them
A TPU is an application-specific integrated circuit (ASIC) Google designed to run one class of computation exceptionally well: the dense matrix multiplications inside machine learning models. The first TPU went into production inside Google’s data centers in 2015; the externally available versions arrived in 2018. Each generation has made the architecture more capable and more focused.
The current publicly available lineup on Google Cloud includes the TPU v5e (optimized for cost-efficiency on training and inference), the TPU v5p (the highest-throughput option, aimed at large-scale training runs), and Trillium (the sixth-generation TPU, also referred to as TPU v6e, which Google announced in 2024 as delivering roughly 4.7x the peak compute per chip compared to v5e). These chips are available only through Google Cloud; you cannot buy or rent them anywhere else, and you cannot run them on-premise.
Each TPU chip connects to a high-bandwidth memory subsystem and is designed to be used as part of a TPU Pod, a tightly interconnected cluster that scales from a handful of chips to thousands. The inter-chip interconnect bandwidth inside a Pod is substantially higher than what you get connecting separate GPU servers over InfiniBand or Ethernet, which is the architectural reason TPUs perform well on workloads that require frequent synchronization across many accelerators.
The Core Architectural Difference: Systolic Arrays vs. General Parallelism
The deepest difference between a TPU and a GPU is not clock speed or memory bandwidth. It is how each chip handles matrix multiplication.
A GPU achieves parallelism by running thousands of independent threads simultaneously. Each thread can execute arbitrary code, which makes the GPU adaptable but also means the chip spends transistor budget on branch prediction, thread scheduling, and general memory access patterns. Tensor Cores narrow this somewhat by fusing multiply-add operations, but the GPU still has to orchestrate those Tensor Cores through a general-purpose scheduler.
A TPU uses a systolic array: a grid of multiply-accumulate units arranged so that data flows through the array in a wave pattern, with each unit passing its partial result directly to the next. There is no general scheduler managing this flow. The result is that a TPU can sustain extremely high arithmetic throughput on matrix operations with very little overhead, because the hardware is essentially hardwired for that computation. The tradeoff is inflexibility. If your workload does not map cleanly to large matrix multiplications, the systolic array sits partially idle and your effective utilization drops.
This architectural gap explains why TPUs tend to outperform GPUs on the specific workload they were designed for, and why they underperform on irregular operations like dynamic computation graphs, sparse attention patterns, or custom CUDA kernels with no TPU equivalent.
Performance: Where Each Accelerator Actually Wins
For large-scale training of dense transformer models, TPUs have consistently shown competitive or superior throughput per dollar compared to equivalent GPU configurations, particularly when the model fits the systolic array’s preferred shapes. Google trained the PaLM family and many of its internal Gemini model versions on TPU Pods. The tight interconnect inside a Pod means gradient synchronization during distributed training adds less latency than a comparable GPU cluster connected over a network fabric.
For inference, the picture is more nuanced. TPU v5e and Trillium are built with inference cost in mind, and on high-throughput batch inference workloads with predictable request shapes, they can be highly efficient. But inference frequently involves variable sequence lengths, speculative decoding, and complex batching strategies. The rigidity of the systolic array makes adapting to these patterns harder, and in practice many production inference deployments still run on GPUs, partly because the tooling for serving (vLLM, TensorRT-LLM, TGI) is overwhelmingly GPU-native.
GPUs win decisively on irregular workloads: reinforcement learning with highly dynamic computation graphs, model development and experimentation where you are constantly changing architecture, any workload that relies on custom CUDA extensions, and anything where the batch size is small and latency matters more than throughput. The teams we work with who run real-time, low-latency inference almost uniformly use GPUs.
Cost and Availability: The Portability Tradeoff
GPUs are available across a wide range of providers. You can compare spot pricing across AWS (p4d, p5), Azure (ND H100 v5), Google Cloud (A3), and specialized GPU clouds like CoreWeave. If one provider raises prices or capacity tightens, you have real alternatives. You can also buy NVIDIA H100 or H200 servers outright and run them on-premise, which is attractive when your utilization is high enough to justify the capital expenditure. The analysis of running AI on-premise vs. cloud shows this crossover point is lower than most teams expect.
TPUs are Google Cloud only. Full stop. There is no spot-market alternative, no bare-metal provider offering TPU access, and no on-premise option. You are locked into Google Cloud’s pricing, availability zones, and support model. On a per-chip basis, TPU pricing is competitive with high-end GPU pricing, and Google offers committed use discounts that can reduce costs significantly on long-running training runs. But the moment you need to move workloads for any reason, regulatory, cost, or technical, you are doing a full migration.
The broader question of which cloud platform gives you the best overall AI infrastructure, factoring in TPUs, GPUs, storage, and managed services, is worth examining in depth when choosing a provider; a comparison of AWS, Azure, and Google Cloud AI capabilities covers that tradeoff in detail. The short version: Google Cloud is the only option if you want TPUs, but AWS and Azure typically offer more flexibility and broader GPU availability.
Software Ecosystem: CUDA’s Gravity vs. JAX’s Momentum
The software story matters as much as the hardware specs, possibly more in 2026 when most teams are not writing low-level kernel code themselves.
CUDA is the default. PyTorch, the dominant framework for both research and production ML, runs on CUDA. Hugging Face’s entire model library is built against PyTorch and CUDA. When you pull a pre-trained model from any public repository and want to fine-tune it, the default assumption is that you have a CUDA-capable GPU. The tooling for quantization, serving, profiling, and debugging is overwhelmingly CUDA-centric. This is not going to change quickly; the installed base is too large.
TPUs run best on JAX, Google’s numerical computation library built on the XLA compiler. JAX has genuinely impressive capabilities, including automatic differentiation through arbitrary Python transformations, just-in-time compilation to TPU-optimized code, and a functional programming model that some researchers prefer. TensorFlow also supports TPUs. PyTorch support for TPUs exists via PyTorch/XLA, and the compatibility has improved substantially over the past two years, but as of 2026 it is still not as production-grade as native CUDA PyTorch. You will encounter ops that are not supported, performance gaps on specific model architectures, and a smaller community to debug issues with.
If your team is JAX-native, works primarily with Google’s model ecosystem, or is building something net-new without legacy code dependencies, the TPU software story is workable. If you are running a standard PyTorch-based ML pipeline, migrating to TPUs carries real engineering cost and risk that should factor into your total cost of ownership calculation.
TPU vs GPU at a Glance
This table summarizes the practical differences that drive most accelerator decisions.
| Factor | GPU (NVIDIA) | TPU (Google) |
|---|---|---|
| Architecture | Thousands of general-purpose parallel cores plus Tensor Cores | Systolic array hardwired for matrix multiplication |
| Primary frameworks | PyTorch and TensorFlow on CUDA | JAX and TensorFlow on XLA; PyTorch via PyTorch/XLA |
| Cloud availability | AWS, Azure, Google Cloud, Oracle, CoreWeave, Lambda | Google Cloud only |
| On-premise option | Yes, buy and run your own servers | No, cloud-locked |
| Best workload | Mixed training, real-time inference, experimentation, custom kernels | Large-scale dense transformer training at high utilization |
| Ecosystem maturity | Largest, CUDA-native serving and tooling | Smaller, strongest inside Google’s stack |
When to Choose a TPU vs. a GPU
Neither chip is universally better. The right choice depends on your specific situation. Here is a practical breakdown by scenario:
- Choose a TPU when you are running large-scale pretraining or fine-tuning of dense transformer models (language, vision, multimodal) and your team is already on JAX or TensorFlow. The interconnect inside a TPU Pod is genuinely better for synchronous distributed training at scale, and the cost per FLOP can be competitive when you use committed pricing and achieve high utilization.
- Choose a GPU when you need to run PyTorch code, especially any code that uses custom CUDA extensions or relies on the broader Hugging Face or open-source ecosystem. The integration friction alone often outweighs any theoretical TPU performance advantage.
- Choose a GPU when you need multi-cloud flexibility or are evaluating on-premise hardware. If there is any chance you will need to move workloads away from Google Cloud, building your stack on CUDA keeps your options open.
- Choose a TPU when you are already a committed Google Cloud customer with significant GCP spend and want to access TPU capacity through committed use discounts. The economics improve substantially if you are not paying on-demand rates.
- Choose a GPU for real-time inference with strict latency requirements and variable input shapes. The ecosystem for GPU serving (vLLM, TensorRT-LLM) is more mature for these patterns than anything available for TPUs.
- Choose a GPU for experimentation, prototyping, and research, where you need to iterate quickly, try new architectures, and swap in community libraries. The GPU’s general-purpose nature is a feature during development even if it is less efficient at scale.
Before you commit budget to either accelerator, model the full picture: chip pricing, framework migration effort, and whether you want the flexibility to move workloads later. If you are weighing where to run your stack, our comparison of AWS, Azure, and Google Cloud AI capabilities walks through the provider tradeoffs in detail, and the breakdown of NVIDIA H100 vs H200 GPUs helps you size the GPU side of the decision.
Frequently Asked Questions
Is a TPU faster than a GPU?
On large-scale dense matrix operations, such as training transformer models with large batch sizes, a TPU can achieve higher throughput than a comparable GPU because its systolic array architecture reduces scheduling overhead and the Pod interconnect reduces synchronization latency. On irregular workloads, dynamic computation graphs, or small-batch inference, a GPU is typically faster because its general-purpose architecture handles those patterns more efficiently.
Can you use TPUs outside of Google Cloud?
No. TPUs are exclusively available through Google Cloud as either on-demand or reserved instances, and as part of Google’s internal infrastructure. There is no third-party cloud provider offering TPU access, no bare-metal colocation option, and no way to purchase TPUs for private data centers. This is a fundamental architectural constraint, not a policy that is likely to change.
Are TPUs cheaper than GPUs?
On a per-operation basis for the workloads TPUs are optimized for, they can be cost-competitive with high-end data-center GPUs. However, the comparison is misleading without context. TPUs require Google Cloud lock-in, JAX or TensorFlow expertise, and migration investment. GPU pricing spans a wide range across providers and spot markets. Total cost of ownership, including engineering effort, is usually higher for TPUs unless you are running at scale with high utilization.
Do TPUs work with PyTorch?
PyTorch/XLA provides TPU support for PyTorch, and compatibility has improved significantly since 2022. You can run many standard PyTorch models on TPUs without rewriting them. However, not all PyTorch operations are supported natively, some custom CUDA extensions have no TPU equivalent, and debugging performance issues requires familiarity with XLA compilation. For most PyTorch users, staying on GPU is the lower-risk path.
TPU vs GPU vs CPU: what is the difference?
A CPU executes sequential logic with a small set of powerful cores. A GPU runs thousands of smaller cores in parallel across floating-point operations. A TPU is an ASIC built around a systolic array for the matrix multiplications inside neural networks. In practice: CPUs handle orchestration, GPUs handle most AI workloads, and TPUs handle large-scale Google-ecosystem training runs where JAX or TensorFlow is the framework.