Features of prelude-2
Inside the model, every token is a list of numbers that mix many things at once. A sparse autoencoder, trained on the residual stream after block 4, splits it into 4,096 directions, few of them active at a time: each one tends to mean one thing, like “the last digit of a number” or “a story begins”. Shaded tokens show where a feature fires.
- Features
- 4,096
- Loss recovered
- 98.9%
- Variance unexplained
- 7.3%
- Model loss, with the features
- 2.565 → 2.631
4,096 features.
-
#3098
Fires on “apple”agent 0.46% of tokens>>> repeat("house", 3)⏎"househousehouse
-
#3099
Fires on “rite”agent 0.93% of tokenss only the even numbers.<|user_end|><|assistant_start|><|tool_call_start|>write<|arg|>path
-
#3100
Fires on “upper”agent 1.27% of tokensupper case, but it has `return s.lower()`.⏎
-
#3101
Fires on “ `”python 0.47% of tokens. They make noises and pretend to drive them
-
#3102
Fires on “3”worksheet 1.48% of tokens71 for me?⏎61178+4
-
#3103
Fires on “ "”story 0.64% of tokensthe middle of the room. His mum said, "We can
-
#3104
Fires on “ave”story 0.98% of tokensYou can mix them and make new colors. Have fun,
-
#3105
Fires on “ mark”agent 0.54% of tokens⏎Bug: `exclaim` should add an exclamation mark to s
-
#3106
Fires on “ water”story 0.85% of tokens. They saw many things in the water, like fish and shells
-
#3107
Fires on “y”story 4.44% of tokenstime, there was a little spark named Sparky. Sp
-
#3108
Fires on “ their”story 0.44% of tokensnear their house.
-
#3109
Fires on “-”agent 0.33% of tokens<|tool_start|>Read file mathlib.py, lines 1-9⏎
-
#3110
Fires on “ time”chat 0.51% of tokenslots of fun. After a while, it was time to go
-
#3111
Fires on “ three”story 0.10% of tokensthey played the game.<|assistant_end|><|bos|>Johnny was three. He
-
#3112
Fires on “.”story 0.95% of tokens, don't go in there. It's dangerous. Mom will
-
#3113
Fires on “ he”story 10.64% of tokensgirl who loved to play outside. One day, she decided to
-
#3114
Fires on “⏎”story 0.99% of tokenshe was so interested in learning.⏎Dav
-
#3115
Fires on “ cried”story 1.12% of tokensflood!" she said. Fluffy meowed and hid under the
-
#3116
Fires on “ the”agent 0.58% of tokensand the bird was very proud. He flew around the world,
-
#3117
Fires on “4”worksheet 1.33% of tokens1 for me?⏎0002+133
-
#3118
Fires on “⏎”worksheet 1.19% of tokens4?⏎86728+000
-
#3119
Fires on “ we”python 0.03% of tokenslike a good plan, but the turtle reminded them all that
-
#3120
Fires on “ ”python 0.14% of tokens>>> how_many([5, 4, 4,
-
#3121
Fires on “3”chat 0.45% of tokens2))⏎print(repeat("music", 2))⏎
-
#3122
Fires on “ How”python 0.64% of tokens.<|bos|><|user_start|>hey there<|user_end|><|assistant_start|>Hello! How can I
-
#3123
Fires on “ his”worksheet 1.36% of tokensfriendly dog came up to him. The dog wagged its tail and
-
#3124
Fires on “.”worksheet 0.68% of tokens96 and 8 is 104.⏎Ben
-
#3125
Fires on “So”worksheet 0.78% of tokens=8, 88009⏎So 8
-
#3126
Fires on “ a”story 0.35% of tokensa story about a car?<|user_end|><|assistant_start|>Once upon a time,
-
#3127
Fires on “ ”worksheet 2.21% of tokensFor example, `raise_to(3, 1)`
-
#3128
Fires on “ a”story 0.49% of tokensout a big yawn and decided to take a nap.
-
#3129
Fires on “ soon”agent 0.62% of tokensis what was behind the door!" she exclaimed in delight.
-
#3130
Fires on “ toys”worksheet 0.45% of tokensBen gets 80 more toys. How many toys does Ben
-
#3131
Fires on “⏎ ”python 0.12% of tokens== 8⏎assert decrement(1) == 0⏎
-
#3132
Fires on “ heard”story 1.85% of tokenstheir trucks or play together.⏎Then, they heard a loud
-
#3133
Fires on “py”agent 0.47% of tokensa coding agent harness.<|system_end|><|user_start|>Write strings.py with a
-
#3134
Fires on “ for”story 0.95% of tokensbig tree."⏎They played near the big tree for a long
-
#3135
Fires on “ gets”chat 0.17% of tokens.<|system_end|><|user_start|>Mia has 8 stickers. Mia gets 2
-
#3136
Fires on “7”worksheet 0.21% of tokens183⏎
-
#3137
Fires on “ no”story 0.05% of tokenssteam in the air and knew that there was no more fun
-
#3138
Fires on “music”agent 0.61% of tokens3, in <module>⏎ assert combine("tiger", "
-
#3139
Fires on “ trained”agent 0.66% of tokens<|assistant_start|>I'm prelude, a very small language model trained from scratch
-
#3140
Fires on “tools”agent 0.42% of tokensto tell stories and add numbers.<|assistant_end|><|bos|><|system_start|>tools: edit
-
#3141
Fires on “ lived”story 0.51% of tokensto play outside. She had a bird friend who lived in a
-
#3142
Fires on “ The”story 2.01% of tokensbye to their friends and went home. The loud party
-
#3143
Fires on “6”story 0.54% of tokens6 +
-
#3144
Fires on “ has”python 0.28% of tokensone` should return x plus one, but it has `return
-
#3145
Fires on “.”agent 1.12% of tokens8]⏎22⏎<|bos|># utils.py⏎
-
#3146
Fires on “=”story 0.33% of tokens2+3: 2+3+1=6,
-
#3147
Fires on “9”worksheet 1.15% of tokens384⏎So 44832 +