Machine Learning from Scratch: Seven Decisions Behind a Model

Re-edited on July 30, 2026, this article replaces an encyclopedia with a practical thread: machine learning fits a system to data in order to improve a defined measure. It does not learn in the abstract, always discover “hidden patterns,” or remove the need to program a task. People choose the example, signal, representation, objective, model, test, and use.

Imagine predicting whether an order will arrive late. Before trying a network, define what counts as an order, when the prediction occurs, which information existed at that moment, who sees the output, and what each error costs. Those choices matter more than the algorithm’s name. The durable skill is reconstructing them in any project.

Examples, variables, objective: the problem contract

A row is not always an example. It may represent a person, visit, machine, transaction, or time window. If one person appears in both training and testing, a model may recognize them instead of generalizing. The unit must match what will be new when the system is used.

Input variables, or features, may contain only information available at decision time. Including an invoice issued after a delay leaks the future. Provenance, missingness, frequency, and transformation are documented. A missing value may mean “not measured,” “not applicable,” or sensor failure; turning all three into zero creates a fiction.

The objective translates a need into a number that training tries to improve. Classification estimates classes or probabilities; regression, quantities; ranking, order; generation, a distribution of outputs. A loss function is a mathematical surrogate rather than the social purpose. Minimizing average error does not guarantee fairness, safety, or usefulness.

A baseline may predict the frequent class, use a rule, or preserve the existing process. Without one, a number does not show whether a model helps. The test set is reserved before experimentation and represents the future population and period. Validation supports selection; testing supports final estimation.

Prediction is not causation. A variable that anticipates delay may not prevent it when changed; it may indicate another cause or be an early consequence. If a decision is intended to intervene, state which action follows the score and study its effects. A predictive model may prioritize cases without answering which policy improves outcomes.

Supervised, unsupervised, and reinforcement describe the signal

In supervised learning, each example has a target output. Labels may come from measurements, decisions, or annotators and include their errors. “No disease” may mean no follow-up occurred; a return may arrive later. Teams measure agreement, delay, and who was excluded.

Classification and regression can use many families. Support-vector machines seek boundaries with margin properties; trees partition space through rules. XGBoost implemented regularized boosted trees with system optimizations. No method always wins: data, metric, and constraints decide.

Unsupervised learning lacks an equivalent target output and seeks clusters, density, factors, or representations. A cluster is not a natural category; it depends on distance, scale, group count, and sample. It can support exploration or compression, but assigning human names requires external evidence.

In reinforcement learning, an agent acts, receives rewards, and affects future states. DQN combined Q-learning and deep networks in Atari games under a defined environment. Reward, observability, actions, and resets make up the task. Mastering a game does not demonstrate general judgment or authorize real actions without simulation and controls.

Model, parameters, and optimization

A model defines a family of functions; parameters select one member. Linear regression combines variables; a tree creates partitions; a network composes transformations in layers. “Neural” is a historical and mathematical analogy, not evidence that a network emulates a brain.

During training, an optimizer changes parameters to reduce loss. Stochastic descent estimates gradients with batches. Adam combines adaptive estimates of first and second moments. It may converge quickly on certain problems, but it does not guarantee the best minimum or generalization. Learning rate, batch, initialization, and schedule are recorded.

An activation such as ReLU adds nonlinearity; layers allow hierarchical representations. More depth increases capacity and optimization difficulty, not knowledge by definition. Architecture embeds biases: convolution favors locality, attention relates positions, and trees handle thresholds. A useful bias can reduce data and cost.

Generative models use objectives to learn distributions or produce samples. GANs train a generator and discriminator in a game; they are neither a universal “future direction” nor a realism guarantee. A convincing sample and a correct prediction require different tests.

Generalization: learn the regularity, not the exam

Overfitting means a system performs better on data used to fit it than on relevant new cases. It is not limited to memorizing rows: a model may learn artifacts of a hospital, camera, date, or annotator. Splits follow the source that will be new and, when time matters, test on a later period.

Regularization changes training or capacity. The dropout paper describes randomly omitting units during training to reduce co-adaptation and approximate combining subnetworks. It does not “prevent extremely high weights”; weight penalties address that more directly. Dropout does not replace a separate test either.

Early stopping uses validation; L1 or L2 penalize parameters; data augmentation creates transformations that must preserve the label. Every technique embeds assumptions. Flipping an image may be valid for an object and false for a letter. Appropriate regularization improves future data under the same evaluation rules.

Bias and variance support diagnosis. A model that is too simple fails even on training data; a flexible one may vary across samples. Learning curves compare performance with data quantity. If both curves are poor, more examples may not repair a wrong objective; if the gap is large, data or regularization may help.

A probability needs a threshold before it becomes an action. That threshold trades false positives against false negatives and may change with review capacity rather than retrospective convenience. Calibration is also tested: among cases scored near 0.8, does the event occur about eight times in ten? Accurate classifications with poorly calibrated probabilities can misallocate resources.

Error analysis groups failures by possible cause: ambiguous labels, missing data, subgroup, time shift, or shortcut. Preserve examples and change one component before repeating the test. Looking only at an average invites complexity without identifying the repaired problem. Sometimes the best improvement is changing collection or the decision rather than the algorithm.

From notebook to a system someone can audit

Documentation begins before modeling. Datasheets for Datasets proposes recording motivation, composition, collection, preprocessing, uses, and maintenance. It can reveal an unrepresented population or period and prevents treating a file as a natural fact.

Model cards document intended uses, factors, metrics, evaluation data, and limitations. A service record adds version, dependencies, threshold, owner, oversight, latency, and incident response. A model does not operate alone: a rule or interface can change harm while its metric stays fixed.

The complete flow is: formulate the decision; collect and document data; split sets; build a baseline; train; tune on validation; evaluate once on testing; analyze errors; pilot in shadow mode; monitor; and retire. Repeatedly looking at test data to improve turns it into validation and requires another test.

In production, populations, sensors, policies, and behavior change. Teams monitor inputs, performance with delayed labels, subgroups, abstentions, corrections, cost, and consequences. Drift triggers diagnosis before retraining. The model may remain stable while the meaning of a label changes.

The transferable skill is completing seven fields for any project: example, signal, representation, objective, model, test, and use. Names such as SVM, network, Adam, DQN, or GAN occupy only the mechanism field. Machine learning becomes understandable when someone can explain what evidence changes the parameters and which observation would show that the system no longer works.