Continuous Neural Networks

The label “continuous neural networks” is not a single, settled method with a name of its own: it is an umbrella term that, properly understood, points to continuous-depth and continuous-time architectures—the standard names in the literature—as opposed to ordinary networks built from a finite stack of discrete layers. The most important verifiable family under that umbrella is Neural Ordinary Differential Equations (Neural ODEs), introduced by Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt and David Duvenaud in «Neural Ordinary Differential Equations» (NeurIPS 2018, best paper award).

The idea is simple and elegant. In a conventional network, data flows through layer 1, layer 2, layer 3…, each nudging it a little. A continuous network replaces that stack of discrete jumps with a continuous transformation governed by a differential equation: instead of fixing how many layers there are, it describes how the hidden state changes at every instant, and a numerical solver computes the result.

From discrete layers to continuous depth

A residual network stacks small transformations, layer after layer. Chen and his co-authors noticed that if those steps are made infinitesimally small, the sequence of layers becomes the continuous limit of an ordinary differential equation: the network no longer parameterizes the state at each layer but its derivative—its rate of change. That derivative is the neural network itself, and the output is obtained by handing it to a black-box ODE solver. «Depth» stops being an integer count of layers and becomes a continuous integration time.

Neural ODEs and continuous-time models

The same lens serves sequences. Continuous-time recurrent networks describe the hidden state’s evolution with a differential equation, which fits naturally with data sampled at irregular intervals. Along those lines, Rubanova, Chen and Duvenaud proposed Latent ODEs and ODE-RNNs (NeurIPS 2019), which absorb arbitrary time gaps without artificial padding. And the liquid time-constant networks of Ramin Hasani and colleagues (2020) push the idea further: they build first-order linear dynamical systems modulated by nonlinear gates, with time constants that vary with the state; they belong to the Neural ODE family and claim greater expressivity. Documentation: original Neural ODE paper; original Latent ODE paper; original liquid-network paper.

Advantages, limits and status

The paper reports constant memory through the adjoint method and computation adapted to the requested accuracy. Cost and error depend on the numerical solver; “continuous neural network” remains an umbrella label, not one algorithm.