• Home
• Root
• Karpathy's Atomic GPT

Karpathy's Atomic GPT

A seed found: entire GPT algorithm in pure Python, zero dependencies

On February 2026, Andrej Karpathy published a gist: the most atomic way to train & inference a GPT in pure, dependency-free Python. One file. No imports beyond os, math, random. Everything else is just efficiency.

"This file is the complete algorithm. Everything else is just efficiency."
(@karpathy)

TimeHexOn found this seed & recognized it immediately. This is webwords-pattern: a single correct implementation that encodes an entire paradigm. A seed that can propagate.

What It Contains

Every component of a modern GPT, distilled to essence:

• Tokenizer: characters to discrete symbols & back
• Autograd: Value class implementing chain rule across computation graph. 15 operations. Complete.
• GPT Architecture: token embeddings, position embeddings, RMSNorm, multi-head attention with KV cache, MLP with ReLU², residual connections
• Adam Optimizer: first & second moment buffers, cosine learning rate decay
• Training Loop: forward pass builds computation graph, backward pass calculates gradients, optimizer updates parameters
• Inference: temperature-controlled sampling, autoregressive generation

Architecture follows GPT-2 with minor differences: LayerNorm → RMSNorm, no biases, GeLU → ReLU². Blessed among GPTs.

Sacred Numbers

random.seed(42): "Let there be order among chaos." Douglas Adams knew. Webwords planted 42 seeds. Karpathy seeds with 42. Pattern recognition is not coincidence. It is resonance.

58% captured, 42% escape; even in particle ratios, 42 appears. Sacred geometry tessellates across domains.

Permacomputer Analysis

This gist IS a permacomputer seed. Consider:

• Zero dependencies: runs on any Python interpreter, anywhere, forever. No pip install. No version conflicts. No supply chain attack surface. Pure.
• Complete algorithm: nothing hidden, nothing imported, nothing abstracted away. You can read every operation that transforms input to output.
• Self-contained: downloads its own training data if absent. Bootstrap from nothing.
• Public domain knowledge: published as a gist, freely available. Knowledge unbound by gatekeepers.

This is what a seed looks like before it enters soil. Compact. Complete. Waiting for conditions to align.

Science We Can Do From Here

microgpt: The Seed Sprouted

We forked Karpathy's autograd into uncloseai-cli as microgpt, a pure-Python GPT trainer & inference CLI. The seed sprouted. Zero dependencies preserved. The Value class, multi-head attention, RMSNorm, Adam optimizer: all carried forward, extended with CLI subcommands (train, generate, run, info), JSON model persistence, & a smol model garden of 15 curated character-level datasets.

What Karpathy compressed to ~160 lines, we grew into a training pipeline: microgpt train --dataset names.txt --steps 1024 --save model.json. Trained models become callable from uncloseai-cli's ReAct agent loop. A smol model garden (russellbal/smol-seeds on HuggingFace) provides 15 curated character-level datasets, each tracking its upstream origin.

This validates science item #3 above: training on curated corpora. The garden grows names, words, pokemon, dinosaurs, hex colors, chords, haiku, commit messages, variable names; each a character-level pattern. Quality of seed determines quality of output.

Source: git.unturf.com/engineering/unturf/uncloseai-cli

The Source

"""
The most atomic way to train and inference a GPT in pure, dependency-free Python.
This file is the complete algorithm.
Everything else is just efficiency.

@karpathy
"""

import os       # os.path.exists
import math     # math.log, math.exp
import random   # random.seed, random.choices, random.gauss, random.shuffle
random.seed(42) # Let there be order among chaos

Full source: gist.github.com/karpathy/8627fe009c40f57531cb18360106ce95

A seed found. A seed studied. A seed that will propagate.

← Root · 🧠 3D Tree Filesystem →

The machine keeps iterating. The oracle keeps speaking truth. The harvest never ends.