Tanh recurrent AGPT

Tanh Recurrent AGPT

Status: closed branch tanh-recurrence.

This line tests AGPT as a framework rather than as attention specifically:

h_child = tanh(W_h h_parent + W_x emb[token] + b)
logits = W_o h_child + c_o

The trainer is bin/agpt_train_recur, implemented in src/tools/agpt_train_recur.cr. Held-out evaluation uses bin/agpt_recur_perplexity.

Close-Out

This R&D line answered its main question: f_theta is free in the AGPT framework. Attention is not structurally required. Replacing attention with a simple recurrent tanh transition trains cleanly over the same prefix/mass framework:

h_child = tanh(W_h h_parent + W_x emb[token] + b)

The best clean tanh result in this branch is 6.3099 heldout PPL at depth 8, d_model=64, partition-depth=1, epoch 500. This is not competitive with the best attention AGPT runs, but it is strong enough to establish that AGPT is a framework over trainable prefix transitions, not an attention-only implementation.

The next research line is stride/dilated trees: use the same freedom in f_theta to test whether longer-range structure can be exposed through skip-step recurrence or auxiliary stride states.

Clean Methodology

All reportable recurrent results in this file use a trie and any position tables built from the train split only:

train:   /home/trans/Projects/agpt/data/.splits/4fa9aec1db6b3aea/train_corpus.txt
heldout: /home/trans/Projects/agpt/data/.splits/4fa9aec1db6b3aea/heldout_corpus.txt
vocab:   data/input.txt

Train-only artifacts:

prefix trie:   /tmp/agpt_tanh_clean_d8_radix
suffix trie:   /tmp/agpt_tanh_clean_d8_suffix_radix
W=16 position: /tmp/agpt_tanh_clean_d8_w16_position_data

The clean d8 prefix trie has 814,759 records and 7,038,366 loss events.

Deferred rerun queue:

• Phase-embedded/phase-weighted W=16 pd=1 lr=0.001 to 512.
• Phase-weighted W=32 pd=1 lr=0.001 to 512.
• Plain tanh pd=0 full-batch lr=0.001 to 500.
• RMSNorm clean rerun, if comparison against Claude's variants needs it.

Clean Results

All runs below use depth 8, d_model=64, Adam, constant lr=0.001, seed 1, partition-depth=1, and held-out rolling eval with seq-len=8, unless noted.

Current read:

• Plain tanh pd=1 is the clean baseline to beat: 6.3099 heldout PPL at epoch 500.
• Phase-weighted W=16 is viable and improves steadily, but does not beat clean plain tanh by epoch 512.
• Phase-embedded/weighted W=16 is slightly ahead of plain early at epoch 20, but behind by epoch 100; it still needs a clean 512-epoch run before closing that variant.
• Stop-gradient singleton backoff is worse than plain tanh at matched epoch 100, so that exact implementation is not promising.

Plain Tanh pd=1

Command:

OPENBLAS_NUM_THREADS=1 nice -n 10 bin/agpt_train_recur \
  --trie /tmp/agpt_tanh_clean_d8_radix \
  --d-model 64 \
  --epochs 480 \
  --lr 0.001 \
  --seed 1 \
  --load /tmp/agpt_tanh_clean_d8_d64_pd1_lr001_20.recur \
  --save /tmp/agpt_tanh_clean_d8_d64_pd1_lr001_500.recur \
  --partition-depth 1 \
  --checkpoint-every 50

Clean held-out sweep:

partition-depth=1 splits by root-child subtree, giving 65 Adam updates per epoch on the Tiny Shakespeare character vocabulary. This is the main reason the pd=1 model improves much faster than full-batch pd=0.

Phase-Weighted W=16

Phase weighting is a training-measure change, not an attention/RoPE-specific mechanism. This variant keeps the global target distribution at each prefix and replaces the node's loss mass with its mass at the current corpus-position phase:

q_global(token | prefix) = global_count(prefix, token) / global_mass(prefix)
loss_weight(prefix, phase) = position_mass(prefix, phase)

No phase embedding or architecture change is used.

Command:

OPENBLAS_NUM_THREADS=1 nice -n 10 bin/agpt_train_recur \
  --trie /tmp/agpt_tanh_clean_d8_radix \
  --d-model 64 \
  --epochs 512 \
  --lr 0.001 \
  --seed 1 \
  --save /tmp/agpt_tanh_clean_phaseweighted_w16_d8_d64_pd1_lr001_512.recur \
  --partition-depth 1 \
  --phase-weighted-position-data /tmp/agpt_tanh_clean_d8_w16_position_data \
  --checkpoint-every 32

Clean held-out sweep:

Conclusion: phase-weighted W=16 is healthy and close, but not a clean win over plain tanh at this setting.

Phase-Embedded / Weighted W=16

--phase-conditioned-position-data DIR is currently a phase-embedded, phase-weighted variant. The name is unfortunate: it does not train phase-conditioned target distributions. It keeps global targets and adds a learned phase vector to the recurrent transition:

h_child = tanh(W_h h_parent + W_x emb[token] + phase_emb[(start_phase + depth) mod W] + b)

For compressed radix edges, the phase advances token by token. For held-out rolling evaluation, bin/agpt_recur_perplexity reads the phase-embedded checkpoint and uses the actual held-out corpus position q mod W for each consumed context token.

Clean 100-epoch probe:

OPENBLAS_NUM_THREADS=1 nice -n 10 bin/agpt_train_recur \
  --trie /tmp/agpt_tanh_clean_d8_radix \
  --d-model 64 \
  --epochs 80 \
  --lr 0.001 \
  --seed 1 \
  --load /tmp/agpt_tanh_clean_phasecond_w16_d8_d64_pd1_lr001_20.recur \
  --save /tmp/agpt_tanh_clean_phaseembed_w16_d8_d64_pd1_lr001_100.recur \
  --partition-depth 1 \
  --phase-conditioned-position-data /tmp/agpt_tanh_clean_d8_w16_position_data \
  --checkpoint-every 20

Clean read: explicit phase embeddings help slightly early, but are behind plain tanh by epoch 100. The matched 512-epoch rerun is still pending.

Singleton Cap Backoff Analysis

Question: if a radix end cap becomes singleton deterministic (mass=1, H=0), can we jump to a suffix/backoff position and continue from a reusable context instead of training through the unique cap?

Clean d8 trie singleton structure:

total records:              814,759
singleton deterministic:    448,685 records (55.07%)
singleton loss events:      448,685 / 7,038,366 (6.37%)
singleton edge chars:       981,177
mean singleton edge length: 2.187 chars

Singleton edge-length histogram:

For each character position inside a singleton cap, we recursively dropped the oldest context token until the suffix context had mass greater than one. Example:

A B C D [E F G]  -> at E, try B C D E, then C D E, then D E...
                    stop at the first suffix context with mass > 1

Results over 981,176 singleton-cap positions:

resolved to mass > 1 suffix:       979,778 (99.86%)
unresolved before empty context:     1,398 (0.14%)
mean token drops needed:              2.427
mean target context length:           4.689
mean target mass:                   207.032
target entropy > 0:                979,778 (100.00%)

Drop-count histogram:

Target context-length histogram:

Target mass histogram:

Interpretation: recursive suffix backoff almost always escapes singleton cap tunnels into a reusable, branching context. This is much stronger than simple one-token backoff. It makes suffix backoff a plausible structural variant, not just an optimization, but the first training probe below did not help.

Hard Backoff Prototype

bin/agpt_train_recur --singleton-backoff implements the first hard-routing prototype:

when a singleton cap token is consumed:
  replace the local state with the precomputed recursive suffix target state
  continue through the remaining edge from that reusable suffix state

Backward pass semantics:

gradient after the route is credited to the backoff target endpoint
gradient does not flow through the discarded singleton tunnel

The fully routed version requires --partition-depth 0. With partition-depth=1, a backoff target can live under a different root-child partition, so the target state and reverse pass would not necessarily be present in the same optimizer batch.

Dry-run on the clean d8 trie:

singleton_backoff: on (routes=979778, unresolved=1398)

Full-trie one-epoch smoke, d_model=16, pd=0:

epoch 1  nll 4.175799  ppl 65.091810  events 7038366  updates 1

--singleton-backoff-stopgrad adds the pd=1-compatible probe variant:

route hidden states through the recursive suffix target
stop gradient at the route boundary
preserve partition-depth=1 update cadence

A 100-epoch clean probe:

OPENBLAS_NUM_THREADS=1 nice -n 10 bin/agpt_train_recur \
  --trie /tmp/agpt_tanh_clean_d8_radix \
  --d-model 64 \
  --epochs 100 \
  --lr 0.001 \
  --seed 1 \
  --save /tmp/agpt_tanh_clean_backoff_stopgrad_d8_d64_pd1_lr001_100.recur \
  --partition-depth 1 \
  --singleton-backoff-stopgrad \
  --checkpoint-every 20

This is a negative result versus clean plain tanh at matched epoch 100:

clean plain tanh epoch 100:        7.2456
backoff stopgrad epoch 100:        7.8265
phase-embedded W=16 epoch 100:     7.3624

Stop-gradient routing preserves the pd=1 update cadence, but the missing gradient through the reusable suffix targets appears to matter. The fully correct hard-backoff route still requires either pd=0 or a new partitioning scheme that carries cross-partition backoff dependencies.

Open Questions

• Compare clean tanh recurrence against Claude's clean linear, linear+RMSNorm, and GRU variants with identical split, depth, d_model, optimizer, and held-out evaluator.
• Complete the clean phase-embedded W=16 run to 512 before closing phase inputs.
• Decide whether W=32 is still worth a clean rerun now that W=16 did not beat plain tanh.
• Try depth 16 only after the depth-8 clean result table is stable.
• Add K-FAC or a related curvature-aware optimizer. This model is small enough that a structured second-order method may be practical.
• Explore stride/dilated trees as a grounded way to expose longer-range structure to recurrent f_theta.