Modular Neural Networks

A modular neural network is made up of several specialized modules or subnetworks, each responsible for part of the problem, whose outputs are combined to produce the final answer. Its motivation is engineering—decomposing the problem, reusing blocks and gaining efficiency—not “emulating human thought”: modularity draws on the brain's organization only as a design principle.

Concepts worth not mixing up

The term gathers distinct ideas. Modularity proper is an internal property of a single model divided into subnetworks. An ensemble is something else: several independent models that run in parallel and whose predictions are averaged or voted; all of them are activated. A cascade chains modules in series, where the output of one feeds the next. And a mixture of experts (MoE) combines several “experts” with a routing network that, depending on the input, activates only some.

The mixture of experts today

The mixture of experts with sparse routing has become key to scaling large language models, because it decouples the total parameters from the compute per token: adding experts multiplies capacity, but only a few are activated per token, so the cost stays bounded. Shazeer and colleagues introduced the idea in 2017; the Switch Transformer (2021) took it to trillion-parameter models by activating a single expert per token, and recent models such as Mixtral use it to match much larger dense models with a fraction of the active parameters.

Three axes to understand them

Three orthogonal axes that everyday usage confuses are worth separating: the architecture (how the network is divided into modules), the routing (how it is decided which module acts for each input) and the composition (how the outputs are combined: weighted sum, average, voting or chaining).