Feature #1041

Strongest examples

On text the dictionary never trained on. The underlined token is where it fires most.

1. chat 3.22
   <|user_end|><|assistant_start|>Once upon a time, in a cheerful forest, there lived a little bird
2. chat 3.03
   !<|user_end|><|assistant_start|>Once upon a time, in a cheerful forest, there lived a little bird
3. story 3.03
   .<|bos|>Once upon a time, in a big palace, there lived a king and
4. story 2.98
   upon a time, in a big forest, there was a naughty monkey
5. story 2.94
   ?<|assistant_end|><|bos|>Once upon a time, in a tidy little farm, there lived a farmer.
6. story 2.94
   back home!<|bos|>Once upon a time, in a small village, there was an ugly tree
7. story 2.89
   <|bos|>Once upon a time, in a big garden, there was a little orange
8. story 2.86
   matter what.<|bos|>Once upon a time, in a small school, there was a classroom
9. story 2.79
   nice day.<|bos|>Once upon a time, in a small town, there lived a little boy
10. story 2.77
    tests passed⏎<|bos|>Once upon a time, in a jolly town, there lived a kind man
11. story 2.76
    .<|assistant_end|><|bos|>Once upon a time, in a peaceful little town, there was a girl named
12. story 2.76
    5.<|bos|>Once upon a time, in a peaceful little town, there lived a kind b