Features of prelude-2
Inside the model, every token is a list of numbers that mix many things at once. A sparse autoencoder, trained on the residual stream after block 4, splits it into 4,096 directions, few of them active at a time: each one tends to mean one thing, like “the last digit of a number” or “a story begins”. Shaded tokens show where a feature fires.
- Features
- 4,096
- Loss recovered
- 98.9%
- Variance unexplained
- 7.3%
- Model loss, with the features
- 2.565 → 2.631
4,096 features.
-
#3700
Fires on “ and”story 0.43% of tokensof all. Spot gave the toy to Tim, and Tim felt
-
#3701
Fires on “.”python 0.22% of tokensopencode.ai/
-
#3702
Fires on “_”python 0.42% of tokens⏎# test_number_
-
#3703
Fires on “⏎”python 0.64% of tokensx):⏎ return x > 0⏎```⏎⏎Bug
-
#3704
Fires on “ tired”story 0.58% of tokens, Max and Tim went back home. They were tired from their
-
#3705
Fires on “ a”story 0.30% of tokensthe kitchen with her mom. Her mom wore an apron
-
#3706
Fires on “ too”story 0.83% of tokensso proud that
-
#3707
Fires on “5”worksheet 1.39% of tokens+7=10, 09⏎425
-
#3708
Fires on “!"”story 0.68% of tokensis my room! You can't make me move!" Lily shouted
-
#3709
Fires on “ as”story 0.34% of tokens. With a little magic, the crown was as good as
-
#3710
Fires on “(”python 0.70% of tokensreturns `1`.⏎⏎`last_letter(s)`
-
#3711
Fires on “p”agent 1.11% of tokensShe stepped forward and quickly said, "Spin, spin
-
#3712
Fires on “+”worksheet 0.23% of tokens⏎8+0: 8+0=
-
#3713
Fires on “[”agent 0.76% of tokenscontent=def initial(s):⏎ return s[0]
-
#3714
Fires on “ bunny”story 0.23% of tokensa bird.<|user_end|><|assistant_start|>One day, a bossy bird named Bob
-
#3715
Fires on “):”python 0.63% of tokensnap.<|bos|>```python⏎def flip(s):⏎ return
-
#3716
Fires on “ even”story 0.45% of tokensonly_even(nums)` keeps only the even numbers.
-
#3717
Fires on “ more”story 0.37% of tokens." They learned that sharing and playing together is much more fun than
-
#3718
Fires on “ of”story 0.48% of tokensfair with each other. And they became the best of friends.
-
#3719
Fires on “<|tool_call_start|>”agent 0.42% of tokens: return x * x * x<|tool_end|><|assistant_start|><|tool_call_start|>edit
-
#3720
Fires on “ his”story 0.14% of tokensthe bottom.⏎In the store, Tim saw his friend,
-
#3721
Fires on “⏎”agent 3.11% of tokens7))⏎⏎$ python3 run.py⏎False⏎
-
#3722
Fires on “ to”story 1.16% of tokense that he loved very much. He would pretend to chop
-
#3723
Fires on “ python”agent 0.27% of tokens$ python3 main
-
#3724
Fires on “ump”chat 0.44% of tokens<|assistant_end|><|bos|>Anna liked to polish her toy trumpet.
-
#3725
Fires on “ one”story 0.26% of tokensthe hole. He called for help, but no one heard him
-
#3726
Fires on “3”agent 0.30% of tokens33333333333⏎assert
-
#3727
Fires on “2”worksheet 4.63% of tokens⏎assert sum_two(8, 12) ==
-
#3728
Fires on “Edited”agent 1.46% of tokens(next_number(3))<|tool_call_end|><|assistant_end|><|tool_start|>Edited app.
-
#3729
Fires on “ made”story 0.46% of tokensa cup. It made a loud
-
#3730
Fires on “1”worksheet 1.49% of tokensme?<|user_end|><|assistant_start|><|think_start|>763+605:
-
#3731
Fires on “2”worksheet 0.38% of tokens+ 23 = 71.⏎3+4
-
#3732
Fires on “ returns”python 0.30% of tokens<|tool_end|><|assistant_start|>`squared(x)` returns the square
-
#3733
Fires on “'t”story 0.43% of tokens!" The little boy was so excited, he couldn't wait.
-
#3734
Fires on “=”worksheet 0.39% of tokens7+899: 7+9=16
-
#3735
Fires on “⏎”python 1.57% of tokensfinal_letter("music") == "c"⏎assert final
-
#3736
Fires on “⏎”python 0.30% of tokenspy⏎from strings import upper_case, combine⏎⏎assert
-
#3737
Fires on “ Line”agent 0.61% of tokens1: from arith import product, square⏎⏎ Line 3
-
#3738
Fires on “You”agent 0.71% of tokens<|system_start|>tools: calculator<|system_end|><|user_start|>Thank you<|user_end|><|assistant_start|>You're welcome
-
#3739
Fires on “ a”story 3.43% of tokensMia went to the kitchen. She wanted to make a cake.
-
#3740
Fires on “ mom”story 1.23% of tokenssad and didn't like Max.⏎Lily's mom saw that
-
#3741
Fires on “ so”agent 0.33% of tokensshe was really worried. He wanted to help, so he said
-
#3742
Fires on “ went”story 0.50% of tokens.<|bos|>One day, a little girl named Mia went for a
-
#3743
Fires on “ not”chat 0.53% of tokensdone!<|bos|><|system_start|>tools: calculator<|system_end|><|system_start|>Do not use the
-
#3744
Fires on “,”story 0.47% of tokens<|bos|>Once upon a time, there was
-
#3745
Fires on “⏎”python 0.97% of tokens, b):⏎ return a + a⏎```⏎⏎Fix
-
#3746
Fires on “ ”python 0.59% of tokens, bigger⏎⏎assert fact(5) == 12
-
#3747
Fires on “ I”agent 0.17% of tokensabout yourself.<|user_end|><|assistant_start|>My name is prelude. I'm a
-
#3748
Fires on “ harness”agent 1.71% of tokensan AI agent running in OpenCode, a coding agent harness.<|system_end|>
-
#3749
Fires on “⏎”python 0.99% of tokens(s):⏎ return s.lower()⏎⏎⏎def vowels