Feature #2988

Strongest examples

On text the dictionary never trained on. The underlined token is where it fires most.

1. chat 3.08
   trained from scratch on a laptop by mini-lab.<|assistant_end|><|user_start|>Who are you?<|user_end|><|assistant_start|>
2. story 2.87
   Sue was very surprised but happy that her doll could talk. They became best friends and had a
3. chat 2.75
   : I can only tell short stories and add numbers.<|assistant_end|><|user_start|>Who are you?<|user_end|><|assistant_start|>
4. story 2.74
   . Tim was very surprised. He played with the dog and they became best friends. From that
5. chat 2.72
   <|user_end|><|assistant_start|>Goodbye! Have a nice day.<|assistant_end|><|user_start|>Who are you?<|user_end|><|assistant_start|>
6. chat 2.7
   58 = 114928.<|bos|><|user_start|>Who are you?<|user_end|><|assistant_start|>
7. chat 2.69
   8434 = 8500.<|bos|><|user_start|>Who are you?<|user_end|><|assistant_start|>
8. chat 2.66
   and add numbers.<|assistant_end|><|user_start|>Who are you?<|user_end|><|assistant_start|>
9. chat 2.66
   78.<|assistant_end|><|user_start|>Who are you?<|user_end|><|assistant_start|>
10. chat 2.54
    . I can only tell short stories and add numbers.<|assistant_end|><|user_start|>Who are you?<|user_end|><|assistant_start|>
11. chat 2.53
    960.<|assistant_end|><|bos|><|system_start|>tools: calculator<|system_end|><|user_start|>Who are you?<|user_end|><|assistant_start|>
12. chat 2.52
    you like a story, or some numbers to add?<|assistant_end|><|user_start|>Who are you?<|user_end|><|assistant_start|>