What if a language model could learn enough from one pass over a corpus to rival conventional training?
AGPT (Aggregated Gradient Pre-training) explores a different way to organize training: share work across repeated text prefixes, aggregate their evidence, and ask more of each update.
One epoch is an ideal, probably out of reach. This research asks how close we can get.
Text windows often share prefixes. A trie gives those shared beginnings one path, then branches when the text differs.
The same next-token examples, trained two ways.
AGPT represents the shared cas path once. Each branch contributes evidence back through that shared computation.
For a shared prefix, sum descendant gradient contributions before applying the common prefix transformation.
This is an exact algebraic identity. Turning it into a correct, fast transformer training system is the hard part.
Read the paper draftWhen text repeats, a tree can turn that repetition into shared computation and a reusable map of the corpus.
Separate examples may calculate the same prefix state many times. AGPT can share that state across branches and, ideally, compute it once per corpus pass. A node may cost more to process, but fewer repeated evaluations could reduce total work.
Repeated contexts share one path. In Tiny Shakespeare, branching nodes peak at 78,836 at character depth 7, then fall to 6,119 by depth 16. At that cap, 99.43% of depth-16 contexts have one observed continuation: zero empirical next-token entropy. A radix trie folds one-child stretches into edges; other tree structures might compress them further.
On the same 1.1M-character corpus, going from depth 16 to 124 adds just 3.7% more radix node records:
Open scale question: Could branching across all recorded human writing end within roughly 50 tokens? This is a working estimate, still unmeasured. Storage at that scale would depend on how the tails are represented.
By depth 124, Tiny Shakespeare has no repeated character contexts left. Radix edges already compress the long one-child paths; pruning tails or looping back through the root may reduce storage further. Those ideas and any end-to-end speed advantage still need testing. Branching profile ↗ · Depth-124 profile ↗ · Tail pruning ↗ · Root wrap ↗
Shared-prefix training has to find useful context beyond the tree, update often enough to learn, and pay for its extra computation.
Sharing works because short contexts repeat. Long ones rarely do: extend a context far enough and nearly every path through the tree has only one observed continuation. At depth 16 in Tiny Shakespeare, over 99% do. Yet language depends on context far longer than the point where repetition runs out.
Long-range context and prefix sharing in the same model, rather than one traded for the other.
Every branch below a shared prefix feeds one combined step. That is where the savings come from, and also the cost: the weights move once where ordinary training would move them many times. When branches pull in different directions, one combined step can learn less than many smaller ones would have.
Keeping the savings of sharing while learning at least as much per unit of computation as ordinary training.
Ordinary training spends its compute on many small, noisy updates. A shared pass spends it on fewer updates, each built from exact information about the whole corpus. That trade wins only if the exact information is worth more than the updates given up, and only if maintaining the tree costs less than the work it saves: the caches, the gradient through shared prefixes, the structure itself.
Matched model quality in less wall-clock time than conventional training, on the same hardware, counting every cost.
AGPT started as a way to rethink GPT training. Later, the same structure turned out to apply to other models with reusable prefix states.
The original track applies the idea to GPT-style transformers: share repeated token prefixes, combine their gradients, and test whether a corpus pass can carry more useful training signal.
Transformer · Crystal/CUDA · experiment runnerThe tree also works with models that carry a complete state from one token to the next. The PyTorch track explores this with tanh and GRU RNNs, along with ways to combine their training evidence.
Recurrent models · Python/PyTorch · research notebookIf the state after a prefix contains everything needed to process the next token, the same prefix can be computed once and reused across its branches. With differentiable transitions and additive losses, descendant gradients can be combined before passing through the shared computation.
The tracks used different evaluation procedures. Their historical perplexity numbers are documented separately and are not directly comparable.
The code, paper draft, run records, and dead ends are public. Start with the research record and follow the evidence.