Features of prelude-2
Inside the model, every token is a list of numbers that mix many things at once. A sparse autoencoder, trained on the residual stream after block 4, splits it into 4,096 directions, few of them active at a time: each one tends to mean one thing, like “the last digit of a number” or “a story begins”. Shaded tokens show where a feature fires.
- Features
- 4,096
- Loss recovered
- 98.9%
- Variance unexplained
- 7.3%
- Model loss, with the features
- 2.565 → 2.631
4,096 features.
-
#1945
Fires on “That”story 0.66% of tokenstheir hands. They laughed and laughed.⏎"That was fun
-
#1946
Fires on “ to”story 0.47% of tokensa time, there was a little girl who loved to play outside
-
#1947
Fires on “⏎”python 0.89% of tokensinitial(s):⏎ return s[0]⏎⏎#
-
#1948
Fires on “ shell”agent 0.68% of tokensglob, grep, question, read, shell, sk
-
#1950
Fires on “ Thank”agent 0.20% of tokensand beautiful. I improved my drawing. Thank you,
-
#1951
Fires on “1”worksheet 1.05% of tokens6590⏎So 2951 +
-
#1952
Fires on “So”worksheet 0.56% of tokensif she threw
-
#1953
Fires on “ wanted”story 0.51% of tokensto the print shop. She wanted a picture
-
#1954
Fires on “ a”story 0.23% of tokensAnswer in one short sentence.<|system_end|><|user_start|>I want a story about
-
#1955
Fires on “ ”agent 0.46% of tokens3, 7, 7]) == 30
-
#1956
Fires on “=”worksheet 0.55% of tokens0+000: 0+0=0,
-
#1957
Fires on “ new”agent 0.65% of tokensand made a new friend.
-
#1958
Fires on “ don”agent 1.10% of tokensiet<|user_end|><|assistant_start|>Sorry, I don't know
-
#1959
Fires on “ very”story 1.30% of tokens. She loved it and ref
-
#1960
Fires on “.”story 0.30% of tokensin size.⏎"Wow!" said Bill. "I
-
#1961
Fires on “ fun”story 0.97% of tokens. They watched the jets and had a fun day together.
-
#1962
Fires on “ the”story 1.81% of tokenstheir mom said, "I have to go to the store.
-
#1963
Fires on “⏎”python 0.43% of tokens"⏎ ^^^^^^^^^^^^^^^^^^^^^^^^^^^⏎AssertionError⏎⏎Exited with
-
#1964
Fires on “ to”story 0.54% of tokensLilly was happy to discover
-
#1965
Fires on “3”agent 1.04% of tokens3⏎print
-
#1966
Fires on “return”python 0.94% of tokensreturn a *
-
#1967
Fires on “.”agent 0.37% of tokensgreetings.py⏎main.py⏎README.md⏎
-
#1968
Fires on “ day”chat 0.06% of tokenswant a story about the park!<|user_end|><|assistant_start|>One day, a
-
#1969
Fires on “ what”agent 0.27% of tokensevery answer with "Sure!".<|system_end|><|user_start|>Hey, what's
-
#1970
Fires on “ shell”agent 0.56% of tokensglob, grep, question, read, shell, sk
-
#1971
Fires on “ like”story 0.75% of tokenshealthy skipping had helped her win, just like her mom
-
#1972
Fires on “,”story 0.38% of tokensjog near the beach. He saw his friend, a cat
-
#1973
Fires on “ big”story 1.52% of tokensa big, scary monster. It had sharp teeth and claw
-
#1974
Fires on “Read”agent 0.45% of tokensread<|arg|>path=utils.py<|tool_call_end|><|assistant_end|><|tool_start|>Read file utils
-
#1975
Fires on “ l”agent 0.21% of tokensGemma's Mommy was sad that she had given her
-
#1976
Fires on “as”agent 0.17% of tokensling. Suddenly, the lamp fell over and crashed on
-
#1977
Fires on “ a”agent 0.18% of tokensve things on the path. They like to learn new things.
-
#1978
Fires on “ has”agent 0.89% of tokensimport keep_even⏎⏎print(keep
-
#1979
Fires on “ said”story 0.29% of tokensfloor. "We need to clean this up!" said Tom.
-
#1980
Fires on “Once”story 1.20% of tokens<|bos|>Once upon a
-
#1981
Fires on “ what”agent 0.49% of tokensuse the calculator.<|system_end|><|user_start|>Could you tell me what 3
-
#1982
Fires on “ a”agent 0.54% of tokenscount_vowels
-
#1983
Fires on “2”worksheet 1.17% of tokens286
-
#1984
Fires on “,”agent 1.07% of tokensSo, they put in apples, bananas, and other
-
#1985
Fires on “ file”agent 1.02% of tokensreturn x + 1⏎<|tool_call_end|><|assistant_end|><|tool_start|>Created file successfully:
-
#1986
Fires on “They”story 2.10% of tokensYes, yes, yes!" Lily said.⏎They drove for
-
#1987
Fires on “ the”story 0.49% of tokensshe had to stay safe, so she climbed up a tree.
-
#1988
Fires on “ "”python 0.49% of tokens"⏎assert final_letter("river") == "r"
-
#1989
Fires on “=”worksheet 0.43% of tokens+4: 4+4=8,
-
#1990
Fires on “ She”story 0.35% of tokensFluffy inside and showed him to her mom. Her mom said,
-
#1991
Fires on “ch”agent 0.56% of tokens, websearch, write
-
#1992
Fires on “script”agent 1.28% of tokens3]))<|tool_end|><|assistant_start|><|tool_call_start|>edit<|arg|>path=script.py
-
#1993
Fires on “ `”agent 1.06% of tokensreturn every number of nums doubled, but it has `return [
-
#1994
Fires on “What”agent 0.49% of tokens0 = 88009.⏎What is the
-
#1995
Fires on “ than”story 0.03% of tokensmixer. Sue learned that sharing is much better than being selfish