Features of prelude-2
Inside the model, every token is a list of numbers that mix many things at once. A sparse autoencoder, trained on the residual stream after block 4, splits it into 4,096 directions, few of them active at a time: each one tends to mean one thing, like “the last digit of a number” or “a story begins”. Shaded tokens show where a feature fires.
- Features
- 4,096
- Loss recovered
- 98.9%
- Variance unexplained
- 7.3%
- Model loss, with the features
- 2.565 → 2.631
4,096 features.
-
#492
Fires on “ example”worksheet 0.50% of tokensall the numbers from 1 to n. For example, `
-
#493
Fires on “ with”story 0.33% of tokensest cow named Molly. Molly lived on a farm with her friends
-
#494
Fires on “ fish”chat 0.52% of tokens.<|assistant_end|><|bos|><|user_start|>Tell me a story about a fish.<|user_end|>
-
#495
Fires on “ He”story 0.53% of tokensHe liked to
-
#496
Fires on “ tree”story 0.41% of tokensunder a big tree and shared
-
#497
Fires on “ to”agent 0.90% of tokensum said to Jack,
-
#498
Fires on “ she”story 0.33% of tokensknow where he is. She does not know where she is.
-
#499
Fires on “python”agent 1.14% of tokenspython⏎def
-
#500
Fires on “ and”story 0.42% of tokensbecame shiny and clean.⏎After that, Tim and Sam sat
-
#501
Fires on “.”story 0.22% of tokenssmall town, there lived a cute dog named Max. Max was
-
#502
Fires on “Mom”story 0.11% of tokensthe path, Mia spotted something purple.⏎"Mommy, look
-
#503
Fires on “ can”story 0.96% of tokensyou don't always win the game, you can still get re
-
#504
Fires on “ looked”story 0.27% of tokensnear their house. It looked fun and
-
#505
Fires on “ played”story 1.96% of tokensplay in the
-
#506
Fires on “7”worksheet 0.77% of tokens.⏎How much is 97207 +
-
#507
Fires on “Sorry”chat 0.60% of tokenshelp me with my taxes?<|user_end|><|assistant_start|>Sorry, I
-
#508
Fires on “,”agent 1.00% of tokens, read, shell, skill, subag
-
#509
Fires on “Do”agent 0.76% of tokens3, 7, 7]) == 30
-
#510
Fires on “ out”story 0.02% of tokensa clear bag with a rainbow inside. Tom reached out his hand
-
#511
Fires on “s”story 0.37% of tokensmade roads, bridges and hills with the
-
#512
Fires on “'t”story 0.64% of tokensaisy didn't understand what
-
#513
Fires on “ch”agent 0.68% of tokensent, webfetch, we
-
#514
Fires on “ <”agent 0.77% of tokens_lists.py", line 3, in <module>
-
#515
Fires on “ long”story 0.76% of tokensuse the calculator
-
#516
Fires on “<|assistant_end|>”agent 1.09% of tokensnewString=hello<|arg|>replaceAll=true<|tool_call_end|><|assistant_end|><|tool_start|>Edited
-
#517
Fires on “ ”chat 0.51% of tokens8: 8+8=16, 16
-
#518
Fires on “ has”worksheet 0.78% of tokens26618.<|bos|><|user_start|>Lucy has 1
-
#519
Fires on “ It”story 1.07% of tokensheard a voice
-
#520
Fires on “<|bos|>”chat 0.93% of tokensday playing with the treasure in the park.<|assistant_end|><|bos|>Once upon
-
#521
Fires on “Created”agent 0.26% of tokens<|tool_start|>Created file successfully: greeting.py<|tool_end|><|assistant_start|>Created `greet
-
#522
Fires on “=”worksheet 0.60% of tokens49.⏎6 + 810 = 8
-
#523
Fires on “ was”story 0.74% of tokenswas a little girl named Lizzy. She was so excited
-
#524
Fires on “<|user_start|>”chat 0.74% of tokens<|assistant_start|>The answer is 126.<|assistant_end|><|user_start|>Plus
-
#525
Fires on “ was”story 0.75% of tokenswas big and
-
#526
Fires on “ +”worksheet 0.53% of tokens3580.⏎Calculate 19 + 3
-
#527
Fires on “ found”story 0.47% of tokensknew she needed to be quiet. So, she found a way
-
#528
Fires on “ them”story 0.62% of tokensed up being friends, and she even let him look at
-
#529
Fires on “ent”agent 1.33% of tokens, read, shell, skill, subagent, we
-
#530
Fires on “:”story 0.13% of tokens2, 2))⏎ ^^^^^⏎NameError: name '
-
#531
Fires on “Yes”story 0.36% of tokenswalk with you?" The seal said, "Yes, let
-
#532
Fires on “ ”python 0.30% of tokens, 2, 2, 8, 5])
-
#533
Fires on “ very”story 0.47% of tokenswas. It was a shiny coin! Amy was very happy to
-
#534
Fires on “1”worksheet 0.10% of tokens: 7+8=15, 158
-
#535
Fires on “`:”agent 0.48% of tokens`abs_value` in `calc.py`: it used
-
#536
Fires on “`”python 1.37% of tokensreturn a - b⏎```⏎⏎`subtract` subtracts
-
#537
Fires on “)”python 1.32% of tokens, divides⏎⏎assert abs_value(-6) ==
-
#538
Fires on “ who”story 0.40% of tokensfeel better.⏎They went to a kind man who knew how
-
#539
Fires on “(”story 0.26% of tokens(backwards("
-
#540
Fires on “1”worksheet 0.47% of tokens⏎6+9: 6+9+1=1
-
#541
Fires on “ plus”agent 0.29% of tokens<|bos|><|user_start|>Hey, what's 440 plus 7