Features of prelude-2
Inside the model, every token is a list of numbers that mix many things at once. A sparse autoencoder, trained on the residual stream after block 4, splits it into 4,096 directions, few of them active at a time: each one tends to mean one thing, like “the last digit of a number” or “a story begins”. Shaded tokens show where a feature fires.
- Features
- 4,096
- Loss recovered
- 98.9%
- Variance unexplained
- 7.3%
- Model loss, with the features
- 2.565 → 2.631
4,096 features.
-
#1043
Fires on “0”worksheet 0.35% of tokens4?⏎86728+000
-
#1044
Fires on “ so”story 2.64% of tokensabout a magical palace in the woods, so one day
-
#1045
Fires on “⏎”python 0.51% of tokens0:⏎ return -x⏎ return x⏎⏎⏎def divides
-
#1046
Fires on “.”worksheet 1.94% of tokens= 1316.⏎Ben
-
#1047
Fires on “`”agent 0.55% of tokens<|assistant_start|>Renamed `halve` to `my_halve` in `
-
#1048
Fires on “ ”python 1.48% of tokens8, 7, 1]) == 7⏎
-
#1049
Fires on “Lily”story 0.52% of tokens<|assistant_end|><|bos|>Lily loves to
-
#1050
Fires on “,”agent 0.68% of tokens, glob, grep, question, read, shell,
-
#1051
Fires on “+”worksheet 0.83% of tokens07⏎2+3: 2+3+
-
#1052
Fires on “ forgot”story 0.61% of tokens⏎By the end of the day, everyone forgot about the
-
#1053
Fires on “ what”chat 0.68% of tokens7=10, 109<|think_end|>Sure! The
-
#1054
Fires on “ tests”agent 3.08% of tokenswas not a nice friend after all. And that is the end
-
#1055
Fires on “ ”worksheet 0.66% of tokens1545.⏎69 plus 64
-
#1056
Fires on “9”python 0.93% of tokens⏎⏎assert sum_up_to(12) ==
-
#1057
Fires on “ boy”story 0.18% of tokens8.<|assistant_end|><|bos|>One day, a little girl named Lily
-
#1058
Fires on “ lots”story 0.71% of tokenswent outside. The park was very big and had lots of fun
-
#1059
Fires on “ little”story 0.32% of tokens<|bos|>One day, a little ball named
-
#1060
Fires on “1”python 0.37% of tokens5) == 6⏎assert larger(1,
-
#1061
Fires on “ there”python 0.48% of tokenssafe. And from that day on, Lily always made sure her
-
#1062
Fires on “ and”worksheet 0.25% of tokens<|assistant_end|><|bos|>Tim had 7111 toys and found
-
#1063
Fires on “#”agent 0.54% of tokensin nums]⏎⏎# README.md⏎# Sc
-
#1064
Fires on “ talk”story 1.23% of tokensstarted to talk! The
-
#1065
Fires on “1”worksheet 0.61% of tokens+25: 6+5=11,
-
#1066
Fires on “ having”story 0.57% of tokensfun and play. One day, they decided to have a big
-
#1067
Fires on “ the”chat 0.36% of tokens⏎Bug: `total` should add up all the numbers in
-
#1068
Fires on “<|system_end|>”agent 1.16% of tokens<|system_start|>tools: calculator<|system_end|><|system_start|>You
-
#1069
Fires on “ of”story 0.77% of tokensit!" said Ant, and so the two of them lift
-
#1070
Fires on “ to”chat 0.15% of tokens<|assistant_end|><|bos|><|user_start|>thanks<|user_end|><|assistant_start|>Happy to help!
-
#1071
Fires on “,”story 2.02% of tokensowsing the books. He noticed a big, orange book
-
#1072
Fires on “ named”story 1.18% of tokens<|bos|>Once upon a time there was a little girl named Liz
-
#1073
Fires on “ gre”agent 0.34% of tokens.<|bos|><|system_start|>tools: edit, glob, grep,
-
#1074
Fires on “ became”story 1.87% of tokensBut then, the sky got dark and
-
#1075
Fires on “ in”worksheet 0.77% of tokensthe clown and said, "Hello! I wish I could have
-
#1076
Fires on “ going”story 0.40% of tokensSpot and his friends had to go home ear
-
#1077
Fires on “7”agent 0.80% of tokensmax_item([5, 2, 2]))⏎⏎
-
#1078
Fires on “0”agent 0.53% of tokens5⏎assert keep_even([7,
-
#1079
Fires on “⏎”worksheet 0.44% of tokens79 and 14782.⏎00
-
#1080
Fires on “ re”story 0.21% of tokensand thanked Ella for her help. Ella and Billy remained
-
#1081
Fires on “ and”story 0.80% of tokensmighty man named Bill. He was very strong and liked to
-
#1082
Fires on “ not”agent 0.26% of tokens⏎NameError: name 'niitial' is not defined.
-
#1083
Fires on “ day”worksheet 0.47% of tokens. Lily thought it was the most beautiful thing she had ever seen
-
#1084
Fires on “<|bos|>”chat 0.72% of tokensday playing with the treasure in the park.<|assistant_end|><|bos|>Once upon
-
#1085
Fires on “ you”chat 0.22% of tokensa helpful assistant.<|system_end|><|user_start|>Can you add
-
#1086
Fires on “ st”agent 1.49% of tokensHelp! I am scared!" Ben runs down the stairs.
-
#1087
Fires on “ on”story 0.59% of tokensone here somewhere," said Lucy with a smile on her face
-
#1088
Fires on “ a”story 0.54% of tokenstell me a story?<|user_end|><|assistant_start|>Once upon a time,
-
#1089
Fires on “:”python 0.24% of tokens3 == 0⏎```⏎⏎Fix: replace `
-
#1090
Fires on “ <”agent 2.07% of tokenscareful when sailing and to not go out in bad weather
-
#1091
Fires on “ boy”story 0.57% of tokensto do.⏎Then, they saw a man. He had
-
#1092
Fires on “ for”worksheet 0.67% of tokensonly_even(nums):⏎ return [n for n in