Andrej Karpathy leaves OpenAI to focus on personal projects

Andrej Karpathy announced on February 13, 2024 that he had left OpenAI the previous day. He said the departure was not caused by a particular event, issue or drama and that his immediate plan was to work on personal projects. That wording matters: at that date he had not announced an education project, even though his public record as a technical teacher made that interest visible.

An individual departure can invite stories about a company’s strategy, but the available fact was narrower. It also opened a more durable question: Karpathy had built open materials for learning neural networks and language models from their components. Following that route replaces the image of a chatbot as a magic box with a system that can be decomposed, trained and evaluated.

Known, attributed and inferred

Karpathy’s departure announcement is the primary source for his decision. It establishes the date, exit, absence of a specified incident and plan for personal projects. It does not identify those projects. “Leaving to focus on education” would turn an earlier record into a declared motive. A rigorous account keeps three columns: verifiable fact, the person’s explanation and the observer’s inference.

His departure is a fact. That there was no drama is an attribution to him: the source knows his own decision, but readers cannot independently inspect every motive. Predicting that his next work may use his teaching experience is an inference. It may be reasonable, but it needs a label and cannot replace a statement that was never made.

The method transfers to hiring, firing and resignation stories: find the original message; retain only what it supports; list the questions it leaves open; and seek confirmation from the organisation. Lack of evidence for a conflict does not prove nothing happened, but it does not permit invention. “Unknown” is informative when it precisely marks the gap.

Three levels for understanding a model

The first layer is use: write a prompt and observe output. It teaches interface possibilities and limits but little mechanism. The second is inspection: vary input, measure results and read documentation. The third is reconstruction: implement a small version to see its data, objective and failure modes. Reproducing GPT-4 is unnecessary to learn the structure it shares with a tiny model.

Karpathy’s Neural Networks: Zero to Hero follows that staircase. It begins with backpropagation, proceeds through a character model, tensors, train-development-test splits, and reaches a GPT-style transformer and its tokenizer. The stated prerequisites are solid Python programming and introductory mathematics, not industrial infrastructure.

The order matters more than the instructor. Backpropagation shows how a loss changes parameters. A character model teaches next-element prediction without hiding behind a complex vocabulary. Separate splits reveal whether a system memorises training data. The transformer adds attention and context. The tokenizer shows that a model receives identifiers rather than words, and that their boundaries can produce odd behaviour.

A small, reproducible laboratory

The first exercise needs a text corpus that can be inspected and legally redistributed. Preserve an original copy and document its licence. Split it into training and validation before changing decisions in response to results. When the same text is used to learn and evaluate, a low loss can measure memory rather than prediction on new material.

Second, build a baseline simple enough to understand: count which character usually follows another. Record validation loss and save samples under a fixed seed. Then add a small neural network and change one variable at a time. Improvement must appear in a pre-defined metric, not merely in a sentence selected afterwards because it “sounds better.”

Third, introduce a tokenizer. Write sentences containing numbers, accents, spaces, names and code; inspect their pieces and token counts. This reveals why visually similar texts can consume different context and why an unusual word is divided. Tokenisation is a learned stage separate from the predicting model.

Fourth, construct causal attention: every position may use earlier tokens, but not future ones, to predict what follows. Verify the mask with a short visible input. Then stack blocks, add residual connections and train. Drawing each tensor dimension —batch, time and features— prevents code from running while the learner cannot explain its representation.

The nanoGPT repository provides a compact implementation for training or fine-tuning GPT-style models. Its educational value is not a promise to build a commercial competitor on a laptop, but readable training and model code. The repository itself distinguishes a small character experiment from reproducing GPT-2, which demands far more infrastructure.

The evidence notebook

Every experiment records code version, data, seed, parameter count, context length, batches, steps, learning rate and training and validation losses. It also retains samples that were not hand-picked. Without this notebook, changing two parameters and keeping the most attractive text produces a demonstration rather than reproducible learning.

Ask three questions after each run. Did validation improve, or only training? Does the result survive several seeds? Which input types became worse? A failure table —repetition, lost coherence, names and long sequences— teaches more than a gallery of best sentences. If the model repeats its corpus, inspect size, regularisation and the split before celebrating an exact sample.

Small scale also teaches what does not transfer. Commercial systems add more data, compute, post-training, filters, retrieval and tools. A classroom model explains token prediction; it does not reproduce all those processes or reveal another system’s private corpus. Knowing where the analogy ends is part of understanding it.

A test of understanding, not memory

A learner understands when they can predict a change’s effect before running it and explain any discrepancy afterwards. They can anticipate that shorter context loses distant dependencies, excessive learning rates destabilise loss and a narrow corpus limits vocabulary. The goal is not reciting component names but connecting data, computation and behaviour.

Another test is dismantling a commercial claim. If a vendor promises “more context,” ask how much remains usable; if it claims “better reasoning,” request the task set and criterion; if it says a system “learns from your documents,” distinguish retrieval from training. Building a small model supplies the vocabulary for these questions without assigning consciousness to statistical prediction.

The February 13 news was a technical figure’s departure and his stated decision to explore personal projects. Its educational value did not require guessing what they would be. His existing materials already led from a character to a transformer and, most importantly, taught measurement. The lasting skill is twofold: separate facts from inference when reading a career move and separate an interface from its components when studying a language model.

What became known five months later

On July 16, 2024, five months after the departure, Karpathy announced Eureka Labs, an AI-and-education company conceived as an AI-native school. That later fact confirms that one personal project took an educational direction. It does not permit moving the information back to February or presenting it as the motive he declared when leaving.

Putting the date before the context preserves the sequence: February established the departure and personal projects; July revealed the nature of one. This distinction updates a story without rewriting its past. A later outcome can explain what happened next, but only a contemporaneous source can support what someone said, knew or intended at the time.