Genetic Algorithms

Genetic algorithms are a metaheuristic for optimization and search inspired by biological evolution. Rather than solving a problem with a closed-form equation, they keep a population of candidate solutions and let it «evolve» across successive generations, favoring the fittest. John H. Holland published Adaptation in Natural and Artificial Systems in 1975; MIT Press's page for the revised edition describes it as the book that initiated the field of genetic algorithms and presented its theoretical foundations. They belong to evolutionary computation, the family of methods that borrows the machinery of natural selection to solve problems.

Each candidate is encoded as a chromosome: a string of values (its genes) representing the problem's variables. The algorithm need not know the mathematical form of the objective; it only needs to measure how good each solution is.

The loop: population, fitness, selection, crossover and mutation

At the core sits a fitness function that scores every chromosome by its quality against the problem. From there, three operators act in each generation. Selection picks parents in proportion to their fitness: roulette-wheel selection assigns probability proportional to the score, while tournament selection pits small groups against one another and keeps the best. Crossover, or recombination, blends two parents into offspring that inherit fragments of each; Holland described this as recombining «building blocks». Mutation injects a random perturbation into a gene, preserving diversity and preventing stagnation. The new population replaces the old one, and the loop repeats over many generations, until a compute budget runs out or progress stalls. Primary source.

Evaluation cost and convergence

Every iteration requires evaluating the fitness of the whole population, and that evaluation cost is the main bottleneck: a single evaluation can be expensive (a simulation, an experiment, a heavy model) and the algorithm needs a great many of them. Larger populations explore more thoroughly but multiply the expense. Being a stochastic method, a genetic algorithm also offers no guarantee of the global optimum: it converges toward good solutions, not necessarily the best one, and its behavior hinges on hyperparameters —population size, crossover and mutation rates, selection scheme— that usually demand manual tuning and trial. Primary source.

When they fit and when they don't

Genetic algorithms are a sensible choice when the search space is large, riddled with local optima, or when the objective is non-differentiable, discontinuous or noisy, cases where gradient methods struggle. They are not, however, a default tool: when a gradient exists and the function is smooth, gradient-based methods tend to be faster and more precise, and for many specific problems there are specialized algorithms that beat them. They are best seen as a robust, general-purpose option for hard landscapes, not a universal answer. Choosing them is an engineering decision that depends on the structure of the problem, the cost of each evaluation, and how much optimality one is willing to trade for robustness. Primary source.