Features of prelude-2
Inside the model, every token is a list of numbers that mix many things at once. A sparse autoencoder, trained on the residual stream after block 4, splits it into 4,096 directions, few of them active at a time: each one tends to mean one thing, like “the last digit of a number” or “a story begins”. Shaded tokens show where a feature fires.
- Features
- 4,096
- Loss recovered
- 98.9%
- Variance unexplained
- 7.3%
- Model loss, with the features
- 2.565 → 2.631
4,096 features.
-
#3349
Fires on “ too”story 0.30% of tokensto crawl up the chair, but it was too high.
-
#3350
Fires on “ sum”agent 1.05% of tokens<|tool_end|><|assistant_start|><|tool_call_start|>grep<|arg|>pattern=def sum_to
-
#3351
Fires on “_”python 0.20% of tokens`.⏎⏎```python⏎def raise_to(
-
#3352
Fires on “ away”story 0.36% of tokensyard. All of Chicky's friends ran away, but
-
#3353
Fires on “f”agent 0.73% of tokensent, webfetch
-
#3354
Fires on “ g”agent 0.22% of tokensBilly was excited and said, "Yes, I would love to
-
#3355
Fires on “3”python 2.16% of tokens.py⏎[8]⏎4.3333
-
#3356
Fires on “ day”story 0.31% of tokensto prevent further danger.<|bos|>One day, a
-
#3357
Fires on “ girl”story 0.95% of tokens⏎As Tom pulled the wagon, he met a girl named Sue
-
#3358
Fires on “py”python 1.27% of tokens(backwards("star"))⏎⏎# items.py⏎def
-
#3359
Fires on “5”worksheet 0.85% of tokens92239⏎00065+9
-
#3360
Fires on “ and”story 2.26% of tokenstoys and make
-
#3361
Fires on “ on”chat 0.39% of tokensnamed Tim wanted to climb the tree. He put on his red
-
#3362
Fires on “1”worksheet 1.37% of tokens5585.⏎If you add 115
-
#3363
Fires on “ `”agent 0.19% of tokens`text.
-
#3364
Fires on “ sorry”story 1.01% of tokens!" His mom laughed and said, "That's okay, Tim
-
#3365
Fires on “return”python 2.19% of tokensHello, " + name + "?"` with `return "Hello
-
#3366
Fires on “-”chat 1.02% of tokenssubtract(9, 12)` returns `-3`.
-
#3367
Fires on “1”worksheet 0.99% of tokens: 7+8=15, 157
-
#3368
Fires on “ thought”story 0.44% of tokensmake something with the toy razor. She thought and thought
-
#3369
Fires on “ upon”story 0.33% of tokensBen has 935 books.<|bos|>Once upon a time
-
#3370
Fires on “ day”story 0.84% of tokensnamed Lily lived with her mom and dad. One day, Lily
-
#3371
Fires on “5”worksheet 0.73% of tokens+ 0 = 274.⏎51 plus
-
#3372
Fires on “ friends”story 0.98% of tokens1540<|bos|>Tom and Mia were best friends. They
-
#3373
Fires on “That”story 0.39% of tokenslaughed and laughed.⏎"That was fun
-
#3374
Fires on “.”agent 0.32% of tokens<|assistant_end|><|user_start|>Add 1 to it.<|user_end|><|assistant_start|>
-
#3375
Fires on “ day”story 0.71% of tokenswould roll and jump in it all day. One day, Tom
-
#3376
Fires on “ inv”story 0.78% of tokens⏎The bird smiled and nodded. The cat removed his
-
#3377
Fires on “ who”agent 0.61% of tokens13<|assistant_end|><|bos|>Once there was a little boy who was playing
-
#3378
Fires on “6”worksheet 1.12% of tokens6+00005: 6+5
-
#3379
Fires on “,”story 0.41% of tokens<|assistant_end|><|bos|>Once upon a time, there was
-
#3380
Fires on “6”agent 1.36% of tokens⏎3: ⏎4: SCORES = [6,
-
#3381
Fires on “ to”agent 0.30% of tokensto play anymore
-
#3382
Fires on “ `”python 0.18% of tokens.upper()⏎```⏎⏎Fix: replace `return s
-
#3383
Fires on “utils”agent 0.73% of tokens` to `minus` in `math_utils.py
-
#3384
Fires on “ "”agent 0.56% of tokensfrom words import loud⏎⏎TEXT = "robot"
-
#3385
Fires on “ me”python 0.38% of tokenswith the number only.<|system_end|><|user_start|>Could you tell me what
-
#3386
Fires on “Once”story 0.59% of tokenswith `final_letter(s)`.<|assistant_end|><|bos|>Once upon a
-
#3387
Fires on “'m”chat 0.71% of tokenscan you help me?<|user_end|><|assistant_start|>I'm prelude:
-
#3388
Fires on “from”python 0.44% of tokens`.<|assistant_end|><|bos|># test_words.py⏎from words import
-
#3389
Fires on “1”agent 0.85% of tokens36: 9+6+1=16,
-
#3390
Fires on “ to”story 0.54% of tokenscane, Amy. We need to stay inside and be safe
-
#3391
Fires on “ was”story 2.01% of tokensand went back to play, knowing that their secret was now a
-
#3392
Fires on “he”agent 0.81% of tokens<|tool_start|>test_calc.py<|tool_end|><|assistant_start|><|tool_call_start|>shell<|arg|>
-
#3393
Fires on “But”story 0.88% of tokens. They were having a lot of fun.⏎But then Tom
-
#3394
Fires on “ little”story 0.49% of tokenssad he was.⏎"What's wrong, sweetheart
-
#3395
Fires on “=”worksheet 0.80% of tokensgiggled as
-
#3396
Fires on “ name”chat 0.32% of tokensbut it has `return "Hello, " + name + "?"
-
#3397
Fires on “1”python 0.39% of tokens⏎ for i in range(n + 1):⏎
-
#3398
Fires on “ she”story 0.11% of tokensto sleep. She took her baby doll everywhere she went.