v2 static calibration (sampled heldout)

baseline-calibration-v2-static-sampled

Status: closed

Question

Measure static v2 calibration runs with epoch checkpoints, using the standard sampled heldout carve. This line ended as a calibration/optimizer sweep for the current CUDAX depth-16 AGPT trainer.

Protocol

Shared settings: bin/agpt_train_v2, static full-trie training, data/input.txt, sampled heldout 5% x 10 chunks seed 42, init data/seeds/shake-d64L2-h4-dff256-s128-seed42.model, model architecture d_model=64, n_layers=2, n_heads=4, d_ff=256, effective AGPT depth/window 16, warmup-cosine with no warmup, anc_grad=true, chunk_queries=50000, checkpoints at powers of two.

The seed filename's s128 is the checkpoint header sequence length. These runs reconcile that seed checkpoint to effective train.max_depth=16 for AGPT training and evaluation.

Early runs vary partition_depth and optimizer. pd=0 is single-fire training: one optimizer update per full trie epoch. pd=1 fires once per root-child group.

Later d128/L6 runs use init rnd/cudax-section2-progressive/seeds/shake-d128L6-h8-dff512-d16-seed42.model with d_model=128, n_layers=6, n_heads=8, d_ff=512, pd=1, anc_grad=true, sampled heldout, wrapped trie, and the same depth/window 16.

Results

<!-- agpt-experiment-table:start -->

d64/L2 1024-Epoch Followup

The 1024-epoch d64/L2 wrapped run was added after the phase-position runs showed continued improvement at 512 epochs. It clarifies that the standard d64/L2 line also had not reached its heldout floor at 512.

The fixed-token PPL bottom in this run was epoch 768 (4.2290). Rolling byte PPL and bits/byte improved slightly again at epoch 1024, but fixed-token PPL regressed by about 0.0090, so the practical d64/L2 bottom appears to be near 768-1024 epochs for this schedule.

d128/L6 1024-Epoch Attempt

Run 20260603T212104-d128l6-depth16-pd1-adam-lr0010-1024ep-cq25k-wrap revisited the larger d128/L6 linear-mass setup with chunk_queries=25000 and planned checkpoints through 1024 epochs. Training failed inside epoch 506 before the epoch-512 checkpoint:

forward pass failed at epoch=506 unit=44/65 rc=627374 chunk=28/32: non-finite forward loss.

The partial run was evaluated with bin/agpt_experiment --eval-run-dir, which skips missing future checkpoints. It produced checkpoint metrics through epoch 384 only.

Although train loss continued drifting down into the low 1.21 range by epoch 505, heldout PPL had already degraded sharply after the epoch-128 checkpoint. This reinforces the earlier d128/L6 result: the useful checkpoint is early, not late, and the old 4.1705 fixed-token PPL at epoch 128 remains the best result in this line.

Closeout

The best result in this line was the larger d128/L6 linear-mass run at epoch 128:

That run continued to 512 epochs but overfit badly after the epoch-128 checkpoint (fixed_token_ppl=8.3467 at 512). The earlier high-LR d128/L6 run with lr=0.0015 also reached the same neighborhood at epoch 128 (fixed_token_ppl=4.1827) before degrading and later hitting the finite-loss guard failure that motivated the optimizer hardening below.

Log-mass and Adam-stability followups were useful diagnostics but did not move the floor:

Interpretation:

• pd=1 is the viable static CUDAX training mode for this baseline; pd=0 converges much worse.
• Adam beats RMSProp for this setup, and lr=0.0015 was best for d64/L2.
• Wrapping the corpus tail was mildly positive and removed the skewed tail construction issue.
• Progressive growth can approach the static d64/L2 result faster, but did not beat the best static run.
• d128/L6 reaches the best fixed-token PPL near epoch 128, but further training overfits sampled heldout.
• Log mass improves some early fixed-token checkpoints but substantially hurts byte/rolling PPL and does not beat the linear-mass d128/L6 result.
• Adam eps and decoupled weight_decay support are useful stability knobs, but neither overcame the depth-16 context ceiling in this sweep.

The line is closed. The next likely gains are structural rather than another optimizer sweep: larger/effective context, position-aware RoPE using corpus position histograms, or backoff/smoothing behavior closer to KN.

Checkpoints

pd=0

pd=1 RMSProp

pd=1 Adam

pd=1 Adam lr0.0015

pd=1 Adam lr0.0015 256 epochs

pd=1 Adam lr0.0015 256 epochs wrapped trie