prelude-2

Features of prelude-2

Inside the model, every token is a list of numbers that mix many things at once. A sparse autoencoder, trained on the residual stream after block 4, splits it into 4,096 directions, few of them active at a time: each one tends to mean one thing, like “the last digit of a number” or “a story begins”. Shaded tokens show where a feature fires.

Features: 4,096
Loss recovered: 98.9%
Variance unexplained: 7.3%
Model loss, with the features: 2.565 → 2.631

4,096 features.

#1345
Fires on “=”
worksheet 1.49% of tokens

006: 4+6+1=11
#1346
Fires on “ a”
story 0.25% of tokens

out a big yawn and decided to take a nap.
#1347
Fires on “ for”
story 1.42% of tokens

he was wrong. The hole was too small for him.
#1348
Fires on “ does”
agent 0.38% of tokens

OpenCode, a coding agent harness.<|system_end|><|user_start|>Why does main.
#1349
Fires on “ can”
story 0.48% of tokens

said, "I am a magic tree. I can talk!
#1350
Fires on “l”
agent 0.57% of tokens

Are the tests passing?"<|user_end|><|assistant_start|><|tool_call_start|>glob<|arg|>
#1351
Fires on “,”
story 0.90% of tokens

around in the air.⏎"Look, Amy!
#1352
Fires on “=”
agent 0.85% of tokens

in s.<|user_end|><|assistant_start|><|tool_call_start|>write<|arg|>path=strings.
#1353
Fires on “def”
python 0.55% of tokens

2])⏎4⏎<|bos|>```python⏎def odd(
#1354
Fires on “<|tool_end|>”
agent 0.31% of tokens

^⏎AssertionError⏎⏎Exited with code 1<|tool_end|><|assistant_start|><|tool_call_start|>
#1355
Fires on “ They”
agent 0.26% of tokens

They went home
#1356
Fires on “ She”
story 1.52% of tokens

deep breath and laughed.⏎The
#1357
Fires on “ took”
story 0.58% of tokens

run to the ball. They pick it up and take a bite
#1358
Fires on “ lost”
story 0.78% of tokens

a little boy. The boy was crying because he lost his toy
#1359
Fires on “⏎”
agent 1.67% of tokens

⏎1: def final_letter(s):⏎2:
#1360
Fires on “ idea”
story 2.20% of tokens

One day, Ben was playing with his
#1361
Fires on “"”
agent 0.61% of tokens

"⏎assert
#1362
Fires on “ you”
agent 0.67% of tokens

you add 9 and 840, you get
#1363
Fires on “0”
python 0.30% of tokens

, 7, 4]) == [18,
#1364
Fires on “But”
story 0.14% of tokens

with the rock and took turns holding it.⏎But then,
#1365
Fires on “7”
worksheet 1.17% of tokens

and 61 for me?⏎7875+
#1366
Fires on “def”
python 0.30% of tokens

(6)⏎True⏎⏎```python⏎def factorial(
#1367
Fires on “,”
worksheet 0.08% of tokens

If you add 9 and 840, you get
#1368
Fires on “,”
agent 0.58% of tokens

<|bos|><|system_start|>tools: edit, glob, grep
#1369
Fires on “t”
agent 0.36% of tokens

replaceAll=true
#1370
Fires on “ s”
agent 0.48% of tokens

wind suddenly blew and made the trees bend and sway.
#1371
Fires on “<|assistant_start|>”
chat 0.84% of tokens

<|bos|><|user_start|>What should I invest in?<|user_end|><|assistant_start|>Sorry,
#1372
Fires on “ to”
story 1.24% of tokens

noticed that the cube started to move
#1373
Fires on “ we”
story 0.41% of tokens

<|system_end|><|user_start|>Hello!<|user_end|><|assistant_start|>Hi! What should we do in
#1374
Fires on “ and”
chat 0.72% of tokens

to help!<|assistant_end|><|bos|>Mia had 0 coins and found
#1375
Fires on “<|bos|>”
chat 0.68% of tokens

know how to tell stories and add numbers.<|assistant_end|><|bos|><|user_start|>Who
#1376
Fires on “ red”
story 0.63% of tokens

, Tim saw a big bug. The bug was green and had
#1377
Fires on “ ”
worksheet 1.97% of tokens

⏎The sum of 6 and 9 is 15
#1378
Fires on “ flew”
story 0.71% of tokens

the mud.⏎Just then, a big bird flew down and
#1379
Fires on “'m”
chat 0.33% of tokens

Luckily, a soft pile of leaves was below him
#1380
Fires on “ and”
story 0.66% of tokens

mom in the park. She saw a big tree and wanted to
#1381
Fires on “ add”
chat 0.59% of tokens

16.<|assistant_end|><|user_start|>And add 9
#1382
Fires on “ an”
story 0.18% of tokens

food is gone!"⏎Tim had an idea.
#1383
Fires on “3”
worksheet 0.76% of tokens

3218.⏎Lucy had 3329
#1384
Fires on “ a”
story 1.30% of tokens

a small house with his mom, dad, and his old dog
#1385
Fires on “ see”
story 0.35% of tokens

inside. Max was very happy. He wanted to see what was
#1386
Fires on “w”
agent 0.70% of tokens

repeats s n times.<|user_end|><|assistant_start|><|tool_call_start|>write<|arg|>
#1387
Fires on “0”
worksheet 0.60% of tokens

. How many balls does Anna have now?⏎000
#1388
Fires on “⏎ ”
python 1.01% of tokens

```python⏎def count_vowels(s):⏎ k =
#1389
Fires on “⏎”
python 0.31% of tokens

⏎$ python3 main.py⏎code cloud⏎1⏎
#1390
Fires on “ "”
story 0.16% of tokens

not want to work.⏎The train thought, "I don
#1391
Fires on “ a”
story 0.11% of tokens

One day, the bird found a long string with a big kn
#1392
Fires on “2”
agent 0.80% of tokens

9, 2, 7, 2])⏎
#1393
Fires on “,”
python 0.86% of tokens

<|assistant_start|>`combine(a, b)`
#1394
Fires on “ for”
agent 0.36% of tokens

to the house.⏎Bill thanked his neighbor for the inv