Adaptive Growth Neural Networks

The label adaptive growth neural networks groups architectures that do not fix their number of neurons in advance but add units during training as the problem demands. It does not name a single algorithm: the literature includes constructive or growing networks and methods with their own rules. A 2022 paper calls the problem neurogenesis in neural networks: deciding when, where and how to add those units.

The motivation is practical. In a conventional network you must choose the size by hand: too small and it cannot learn; too large and it wastes computation and tends to overfit. Constructive networks flip the problem around: they start minimal and grow only when the error justifies it.

The idea: let the network grow

Rather than picking a topology before seeing the data, these methods begin with a tiny network and watch the error. When learning stalls, a criterion decides to add a neuron or a layer, training resumes, and the cycle repeats. The structure thus adapts to the real difficulty of the task instead of being imposed by guesswork.

From classic methods to today

The family includes methods such as Cascade-Correlation, which starts with a minimal network, adds hidden units one at a time and freezes their input weights; and the adaptive merging and growing algorithm, which adjusts structure through both growth and merging. These are examples of the family, not a closed taxonomy.

Limits and open questions

Growth is not free. The decisive questions remain when to add capacity, where to place it, how to initialise it and when to stop. In 2022, Maile and colleagues described adding neurons as a difficult and understudied problem, separating decisions about when to trigger growth, which layer to change and how to initialise the new unit. Excess growth can reintroduce the overfitting it was meant to avoid, so validation and early stopping remain part of the explanation. The paper proposes specific strategies, not a universal rule for every architecture and dataset.