rihim/icl-selfplay-vs-text-results

Self-play vs. text pretraining: ICL results

These are the full evaluation results from comparing the in-context learning (ICL) of:

• the self-play learners from Self-Play Pretraining with Zero Data, and
• same-size models trained on ordinary web text for the same number of tokens.

Code and write-up: github.com/mihir-s-05/icl-selfplay-vs-text. Text-model checkpoints: rihim/icl-selfplay-vs-text-checkpoints.

What was scored

Model families (the arm column):

arm	What	Seeds	Checkpoints
sp	the paper's self-play learners	4	rounds 0, 256, 512, 1024, 2048, 2816, 8191
up	the paper's universal-prior baseline (random programs)	4	same rounds, sizes up to 6M
dclm	same architecture trained on DCLM text	1–2	0.1B → 17.7B tokens
sp2dclm	self-play checkpoint (round 8191), then DCLM	1	0.05B → 0.5B tokens

Sizes: 100k, 500k, 1M, 3M, 6M and 24M parameters.

Task suites (every task shows m worked examples, then a query; the score is greedy exact match on the next byte or bytes):

• raw: the paper's own harness, byte for byte: random bytes 1–255 with a zero byte between examples. Tasks are reverse string, stack, associative recall, sum, max, min, first, last, plus the paper's extra and control cells.
• printable: the same task families written as text, like \nqd=q (first/last/max/min of two letters), \n37+85=122, \n+a+b-b- → a (stack), letter key→value recall, and a letter cipher.
• words: real English words in 10 categories, like \ncat=t\nmouth=b\nrabbit= → t. Covers classification with 2 or 4 arbitrary labels where the query word is unseen in the examples, lookup of a seen word, and reversing a 3-word phrase.

Multi-byte answers count only if every byte is right. The raw reverse task uses per-byte accuracy, as the paper does.

Layout

results/icl/<arm>/<size>/<checkpoint>.json      raw + printable suites (all arms)
results/icl_words/<arm>/<size>/<checkpoint>.json  word suite (all arms)
results/icl_s1/dclm/<size>/<checkpoint>.json    second DCLM seed (1M, 3M, 6M), all suites
results/report/                                 auto-generated report and figures
training_logs/{dclm,sp2dclm}/<size>/seed-*/     log.jsonl + run.json for every training run
training_logs/lrsweep/lrsweep-<lr>/<size>/seed-0/  the short LR-sweep runs
lr.json                                         LR chosen per size
tables/*.csv                                    tidy tables (also shown in the viewer above)

For self-play and universal-prior checkpoints, <checkpoint> is the self-play round. For DCLM runs it is the token count in millions (e.g. 17723M).

Each result JSON looks like this:

{"meta": {"arm": "dclm", "size": "6M", "tokens": 17723031552, "checkpoints": [...], "bf16": true, ...},
 "results": {"<section>": {"<task|m|...>": {"n_trials": 128, "chance": 0.038,
             "per_seed": {"seed-0/learner_17723M.pth": {"acc": 0.93, "p_correct": 0.71, ...}},
             "ensemble": {...}}}}}

ensemble averages the seeds' probabilities before taking the argmax, which is the paper's convention. It is present when a file scores more than one seed.

The tables:

• headline.csv has one row per (arm, size, tokens, suite, seed), holding the suite's headline mean. The word headline excludes word-order reversal, which every model scores 0 on.
• tasks.csv has the per-task cells behind those means.
• mcurves.csv has accuracy against the number of examples m, for every task.
• valbpb.csv has validation bits/byte over training for the DCLM and warm-start runs.
• sp_bpb.csv has the paper's own zero-shot DCLM bits/byte for the self-play ladder.

How these were made

The raw-suite prompts were checked to be byte-identical to the paper's scripts. Scoring the paper's 24M ensemble reproduces its Fig. 4 to within a mean absolute difference of 0.002. Evaluations ran in bf16 on an A100. The paper's scripts used fp32; the difference was noise-level when checked.