Feature #3937

Strongest examples

On text the dictionary never trained on. The underlined token is where it fires most.

1. chat 5.83
   !<|assistant_end|><|bos|><|user_start|>What's your name?<|user_end|><|assistant_start|>I'm prelude, a very small language
2. chat 5.78
   add numbers.<|assistant_end|><|user_start|>Who are you?<|user_end|><|assistant_start|>I'm prelude, a very small language
3. chat 5.78
   : calculator<|system_end|><|user_start|>What is your name?<|user_end|><|assistant_start|>I'm prelude, a very small language
4. chat 5.75
   ?<|user_end|><|assistant_start|>I'm prelude, a very small language
5. chat 5.74
   story!<|assistant_end|><|user_start|>What's your name?<|user_end|><|assistant_start|>I'm prelude, a very small language
6. chat 5.66
   .<|assistant_end|><|bos|><|user_start|>What's your name?<|user_end|><|assistant_start|>I'm prelude, a very small language
7. chat 5.64
   welcome!<|assistant_end|><|user_start|>What is your name?<|user_end|><|assistant_start|>I'm prelude, a very small language
8. chat 5.62
   're welcome!<|assistant_end|><|user_start|>Who made you?<|user_end|><|assistant_start|>I'm prelude, a very small language
9. chat 5.61
   add numbers.<|assistant_end|><|user_start|>Who are you?<|user_end|><|assistant_start|>I'm prelude, a very small language
10. chat 5.61
    tools: calculator<|system_end|><|user_start|>Who made you?<|user_end|><|assistant_start|>I'm prelude, a very small language
11. chat 5.58
    soon.<|assistant_end|><|bos|><|user_start|>Who made you?<|user_end|><|assistant_start|>I'm prelude, a very small language
12. agent 5.53
    you?<|user_end|><|assistant_start|>I'm prelude, a tiny language model