# Pathways

> Machine learning (ML) architecture for distributed cluster computing by Google Research

**Wikidata**: [Q111460849](https://www.wikidata.org/wiki/Q111460849)  
**Source**: https://4ort.xyz/entity/pathways

## Summary
Pathways is a machine-learning software architecture created by Google Research that lets a single model learn thousands of tasks across many machines at once. It replaces the need to train a separate model for every new problem by routing sub-tasks to the right parts of one large, shared network.

## Key Facts
- Introduced by Google Research in 2021 as a new ML system design.
- Classified as both a machine-learning architecture and a distributed-computing framework.
- Runs on computer clusters so work can be split across hundreds or thousands of accelerators.
- Uses sparse activation: only the model sub-networks needed for a task are turned on, cutting energy and cost.
- Shares parameters across tasks, letting earlier skills speed up later learning without full retraining.
- Built to overcome the “one model, one task” limitation of prior Google systems like GShard and Switch Transformer.

## FAQs
### Q: How is Pathways different from today’s large models?
A: Instead of training a fresh model per task, Pathways trains one extensible network that can specialize on the fly; unused parts stay dormant, so compute scales with the request, not the total size.

### Q: Does Pathways only work inside Google?
A: Public papers describe the architecture, but Google has not released open-source code or outside cloud access, so current use appears limited to internal Google workloads.

### Q: What hardware does Pathways target?
A: The design assumes a cluster of TPUs or GPUs connected by high-bandwidth data-center networks; scheduling logic decides which devices activate for each task.

## Why It Matters
Pathways tackles two chronic problems in modern AI: the explosion of single-use models and the exploding cost of training them. By letting one model accumulate skills across vision, language, and structured-data tasks, Google eliminates redundant training runs and the associated carbon footprint. Sparse activation keeps inference cheap, so a trillion-parameter model can behave like a small one when answering a simple query. The routing layer also makes large-scale research safer: new capabilities can be added without overwriting old ones, reducing catastrophic forgetting. If adopted industry-wide, the approach could consolidate thousands of specialized models into a handful of adaptable networks, cutting cloud bills and deployment complexity for developers everywhere.

## Notable For
- First architecture to combine task-level checkpointing, sparse activation, and asynchronous distributed scheduling in one ML stack.
- Enables 1-to-N task mapping: a single set of weights can concurrently serve vision, speech, and text workloads without duplication.
- Demonstrated 3× improvement in training energy efficiency versus dense models of the same nominal size in early Google benchmarks.
- Designed to support multimodal few-shot learning, letting the same model translate text, caption images, and answer structured queries.

## Body
### Architecture
Pathways treats a model as a directed graph of “cells.” A central scheduler dispatches each inference or training step to the minimal path of cells required, leaving others idle. Inter-cell communication uses Google’s internal “Pathways Runtime” that sits on top of gRPC and RDMA for sub-millisecond coordination across TPUs.

### Sparse Routing
Dynamic routing tables decide which cells activate based on task metadata and learned gating weights. The gating function is trained jointly with the network; it outputs a probability distribution over cells, zeroing out most entries. This yields activation rates below 10 % of total parameters for typical tasks.

### Distributed Execution
Jobs are expressed as a data-flow graph. The runtime shards graphs across machines, inserts forward-backward synchronization barriers, and handles device failure by remapping cells to healthy nodes. Checkpoints are written asynchronously to distributed filesystems every few hundred steps.

### Status
As of 2023, Google has not announced open-source releases or external API access. Internal services such as the production translation system and parts of Google Assistant reportedly run on Pathways-backed models.

## Schema Markup
```json
{
  "@context": "https://schema.org",
  "@type": "Thing",
  "name": "Pathways",
  "description": "Google Research machine-learning architecture for distributed, sparse multi-task computing.",
  "sameAs": ["https://www.wikidata.org/entity/Q1234567890"],
  "additionalType": "SoftwareArchitecture"
}

## References

1. [Source](https://blog.google/technology/ai/introducing-pathways-next-generation-ai-architecture/)