prelude-2

Features of prelude-2

Inside the model, every token is a list of numbers that mix many things at once. A sparse autoencoder, trained on the residual stream after block 4, splits it into 4,096 directions, few of them active at a time: each one tends to mean one thing, like “the last digit of a number” or “a story begins”. Shaded tokens show where a feature fires.

Features: 4,096
Loss recovered: 98.9%
Variance unexplained: 7.3%
Model loss, with the features: 2.565 → 2.631

4,096 features.

#141
Fires on “ worry”
story 1.08% of tokens

. His mom smiled and said, "Don't worry, Tim
#142
Fires on “<|arg|>”
agent 0.10% of tokens

danced. She was very enthusiastic
#143
Fires on “ed”
agent 0.85% of tokens

items[-1]
#144
Fires on “ what”
agent 0.32% of tokens

= 1556.⏎Hey, what's
#145
Fires on “.”
story 1.65% of tokens

best of friends forever.<|bos|><|user_start|>
#146
Fires on “print”
python 0.25% of tokens

from textlib import lower_case, length⏎⏎print(lower
#147
Fires on “#”
worksheet 0.38% of tokens

8))⏎⏎# README.md⏎# Demo
#148
Fires on “7”
worksheet 0.63% of tokens

29 = 84.⏎4325
#149
Fires on “ wanted”
story 0.99% of tokens

He wanted to play
#150
Fires on “'t”
story 0.60% of tokens

Tim. His mom smiled and said, "Don't worry,
#151
Fires on “ other”
chat 0.47% of tokens

he could help his mom and dad talk to each other. From
#152
Fires on “ first”
agent 0.35% of tokens

's go to the park and race," said the first child.
#153
Fires on “ =”
agent 0.97% of tokens

* a` with `return
#154
Fires on “ broke”
story 1.06% of tokens

One day, while playing with the bottle, it broke. Tim
#155
Fires on “ thought”
story 0.45% of tokens

to join them. The fearful insect thought about how
#156
Fires on “ be”
story 0.04% of tokens

⏎⏎$ python3 script.py⏎1⏎30
#157
Fires on “ an”
agent 0.71% of tokens

el Tower?<|user_end|><|assistant_start|>Sorry, I'm a tiny model
#158
Fires on “ the”
story 1.01% of tokens

a trip. He flew his plane up in the sky.
#159
Fires on “=”
agent 0.24% of tokens

replacement)<|tool_end|><|assistant_start|><|tool_call_start|>shell<|arg|>command=python3
#160
Fires on “ crab”
story 2.28% of tokens

unexpected happened. The crab started to talk! The crab said,
#161
Fires on “items”
agent 2.17% of tokens

```python⏎def count_items(items):⏎
#162
Fires on “ ”
agent 2.54% of tokens

return x - 1<|arg|>newString= return x
#163
Fires on “.”
story 1.39% of tokens

. Tim was very curious and wanted to know more.⏎At
#164
Fires on “ became”
story 1.14% of tokens

dog. From that day on, Tom and Max became best friends
#165
Fires on “ation”
story 2.08% of tokens

a very special party. The party had colorful decorations,
#166
Fires on “ do”
story 0.23% of tokens

stories and add numbers.<|assistant_end|><|user_start|>What can you do?<|user_end|>
#167
Fires on “<|system_start|>”
agent 0.85% of tokens

_case(s)`.<|assistant_end|><|bos|><|system_start|>tools:
#168
Fires on “ True”
agent 0.17% of tokens

⏎Bug: `has_item` should return True if x
#169
Fires on “ For”
python 0.17% of tokens

_all` returns every number of nums doubled. For example,
#170
Fires on “us”
agent 0.46% of tokens

, much bigger than the other one. She rushed inside
#171
Fires on “ had”
python 0.19% of tokens

has `if
#172
Fires on “#”
story 0.27% of tokens

in nums]⏎⏎# README.md⏎# Sc
#173
Fires on “ nodded”
story 0.24% of tokens

. Do you want to play with it?" Tim nodded and started
#174
Fires on “ He”
story 2.94% of tokens

had an idea. He took the
#175
Fires on “ coding”
agent 0.41% of tokens

You are an AI agent running in OpenCode, a coding agent harness
#176
Fires on “ started”
story 0.66% of tokens

started to race
#177
Fires on “7”
worksheet 1.60% of tokens

9315.⏎5 plus 89 is
#178
Fires on “read”
agent 0.39% of tokens

the first letter of s please"<|user_end|><|assistant_start|><|tool_call_start|>read<|arg|>path
#179
Fires on “⏎”
python 0.50% of tokens

n * 2 for n in nums]⏎⏎⏎def mean
#180
Fires on “ was”
story 0.07% of tokens

at. He ran very fast. But the wind was too strong
#181
Fires on “ has”
worksheet 5.00% of tokens

40 = 649.⏎Mia has 8
#182
Fires on “ ”
agent 0.68% of tokens

x * 2<|arg|>
#183
Fires on “ home”
story 0.34% of tokens

. After a while, it was time to go home. Lily
#184
Fires on “7”
worksheet 0.31% of tokens

<|assistant_end|><|bos|>Tom has 87274 apples.
#185
Fires on “g”
agent 0.20% of tokens

⏎⏎Exited with code 1<|tool_end|><|assistant_start|><|tool_call_start|>grep
#186
Fires on “ ”
python 0.33% of tokens

assert double_all([6, 5, 9,
#187
Fires on “ tried”
story 0.41% of tokens

. She pulled harder and harder. Tom tried to re
#188
Fires on “+”
worksheet 0.65% of tokens

9+44
#189
Fires on “.”
story 2.13% of tokens

so much fun! Then, something unexpected happened. The balls
#190
Fires on “ two”
worksheet 0.42% of tokens

day.<|bos|>Once upon a time, there were two friends called