mini-1

mini-1 is a 5.8M-parameter GPT trained from scratch by the mini-lab training pipeline (pretrain -> midtrain -> SFT -> RL), in 15.5 min of training on a laptop (Apple M5 Pro, 18 cores). It tells short children's stories and adds numbers, either step by step (the scratchpad is returned as reasoning) or by calling a calculator tool when one is provided. It follows a few system prompts, answers follow-up questions and politely declines anything else.

Model

parameters	5,770,496
layers / heads / width	6 / 4 / 256
context length	256 tokens
vocabulary	4096 (byte-level BPE, digits always split)
architecture	decoder-only transformer, RoPE, RMSNorm, GELU, tied embeddings
checkpoint	runs/small/rl

Training

stage	steps	tokens	wall-clock	device	final loss
pretrain	3,500	28.67M	6.8 min	mps	1.460
midtrain	600	4.92M	1.1 min	mps	1.104
sft	300	1.71M	26 s	mps	0.398
rl	300	0.49M	7.2 min	cpu	-
total		35.79M	15.5 min

Hardware: Apple M5 Pro, 18 cores. RL tokens are the sampled completion tokens trained on.

Data

• Pretraining: the first 100 MB of TinyStoriesV2-GPT4 (roneneldan/TinyStories), with synthetic arithmetic worksheets mixed in (25% of documents): equations, sentences, word problems and worked column additions, operands of 1-5 digits.
• Midtraining (format and skills, at volume): single-turn conversations, no system prompt: addition questions in many phrasings answered with a scratchpad (or a calculator call when tools are enabled), operands of 1-3 digits; story requests answered with TinyStories stories; greetings.
• SFT (behavior): a fixed set of 8,000 conversations: system prompts to obey (number only, no calculator, one sentence, start with "Sure!"), follow-up questions about an earlier answer, identity, polite refusals of out-of-scope requests, and plain conversations.
• RL: addition questions with 1-5-digit operands (including lengths the chat data never showed), plus calculator, instruction and refusal problems; reward 1 when the answer is exactly right and in the requested form.
• Operands of 6 digits are never seen in training (length generalization).

Evaluation

Same fixed-seed eval set for every stage (uv run python -m minilab.eval.run):

stage	ppl	1d	2d	3d	4d	5d	6d*	5d@T=1	tool call	tool ans	story	instr	format	train time
pretrain (base)	6.17	34%	7%	4%	7%	2%	0%	-	-	-	-	-	-	6.8 min
midtrain	6.27	100%	100%	100%	0%	0%	0%	0%	65%	65%	96%	15%	67%	1.1 min
sft	7.12	100%	100%	100%	1%	0%	0%	0%	66%	65%	93%	99%	98%	0.4 min
rl	7.37	100%	100%	100%	100%	91%	0%	89%	97%	96%	87%	98%	100%	7.2 min

* held-out digit counts (length generalization). base = raw-text prompt "a + b =". instr = instruction following (see eval.json).

• ppl: perplexity on held-out TinyStories stories.
• Nd: greedy answers to N-digit additions that are exactly The answer is c. (chat format; the base model gets the raw-text prompt a + b = and must continue with the sum).
• Nd@T=1: the hardest in-distribution digit count, sampling at temperature 1 (the API default).
• tool call: with tools=[calculator], the first turn is a calculator call with a correct expression; tool ans: after the tool result, the final answer is correct.
• story: "Tell me a story about a dog." (15 topics x 3 phrasings) -> the story mentions the topic.
• instr: instruction following, the mean of: system prompts obeyed (number only, no calculator, one sentence, "Sure!"), follow-up questions answered, held-out out-of-scope questions refused, identity, and in-scope requests not refused.
• format: fraction of assistant turns properly ended (and no tool call without tools).

Samples (greedy)

Hi!

Hello! How can I help you today?

Who are you?

I'm mini, a very small language model trained from scratch on a laptop by mini-lab.

Tell me a story about a dog.

Once upon a time, there was a little dog named Spot. Spot loved to play with his ball. One day, he saw a big tree with a lot of leaves. Spot wanted to play with the leaves, but he was too small.
Spot had a plan. He would run and jump in the leaves. He would run and jump in the leaves. Spot was very happy. He would run and jump all day long.
Spot's friend, a little bird named Tim, was a small bird. Tim was a small bird. Tim was a small bird. Tim was a small bird. Tim was a small bird. Tim was happy to have a new friend. They played together all day.

What is the capital of France?

I'm sorry, I can't help with that. I only know how to tell stories and add numbers.

[system] Answer with the number only. / What is 12 + 30?

42

What is 12 + 30? / [assistant] The answer is 42. / And add 25 to that?

The answer is 67.

[system] Answer in one short sentence. / Tell me a story about a cat.

Once upon a time, there was a little dog named Max.

Hello

Hi there! Would you like a story, or some numbers to add?

Chat format

OpenAI-style messages rendered with special tokens (see minilab/tokenizer/chat.py). Addition is answered with a scratchpad inside <|think_start|>...<|think_end|> (column by column, right to left), then The answer is c. With tools: calculator, the model emits {"name": "calculator", "arguments": {"expression": "a + b"}} and answers after the tool result.

Limitations

• A toy: it only knows simple children's stories and addition. Anything else gets a polite refusal at best and nonsense at worst. It has no world knowledge.
• Stories are often repetitive or incoherent after a few sentences; the context is only 256 tokens.
• Addition is reliable only for the operand lengths it was trained on; see the held-out columns.
• English only. Not for any real use.