All training runs / Diakosmos / Chat

Diakosmos in conversation

99,823,934 parameters · bytes, no tokenizer · pretrained on 2 GB of OpenWebText, fine-tuned on conversations · served from a rented RTX 3070

Every reply is one run of Alpha's predictor, compiled to sm_86 machine code with no CUDA runtime. It reads the prompt and writes the reply one byte at a time, carrying the recurrence's state from byte to byte, so a step costs the same however long the conversation is.

Ask something short. Diakosmos is small (100M parameters, trained on 2 GB of web text and then 115 MB of conversations), so expect fluent, short and often wrong answers. Each reply is sampled, so asking again gives another.

How it was made

The same Diakosmos as the OpenWebText run, continued on conversations under its own checkpoint contract.

Pretraining: One pass over 2.01 GB of OpenWebText bytes: 15,360 updates in 24.5 GPU hours and 5.29 kWh on an RTX 3070, to 1.419 bits per byte on held-out text. Its final weights are the “pretrained base” above.
Chat fine-tune: Conversations from HuggingFaceTB/everyday-conversations, HuggingFaceTB/smol-smoltalk (without its code), OpenAssistant/oasst2 (each English tree's best-ranked path) and databricks/dolly-15k, each under 2 KB: 93,125 conversations, 114 MB, packed whole into 2,048-byte windows with OpenWebText the run never read filling the rest (about 9%). 960 updates from the run's final weights with a fresh optimizer state, a fifth of the run's peak learning rate, 50 updates of warmup and a cosine to a tenth. That stage learned SmolTalk's long replies (800 bytes at the median) and seldom ended them, and a 100M byte model loses the thread long before then, so a second stage of 320 updates trained on the 58,588 conversations whose every reply is at most 400 bytes; it ends most replies within a few hundred bytes and stays closer to the question. The weights are public: ajaxdavis/diakosmos.
Chat format: Bytes have no special tokens, so roles are plain text, User: … and Assistant: …, and every reply ends with the byte 0x04 (ASCII's end of transmission, which no text uses), which is where the predictor stops. Conversations longer than 1,280 bytes keep their newest exchanges.
Decoding: GPT-2's predictor's draw, run on the GPU by the same Alpha kernel: bytes whose probability is below a fraction (min-p) of the most likely one are excluded, and the byte is the largest of the logits over the temperature plus Gumbel noise drawn from a counter hash of the seed and step. The page samples at temperature 0.7 and min-p 0.1; Greedy picks the most likely byte every step, so the same message always gets the same reply. GPT-2's repetition penalties are left out: a byte's count says nothing about a word's. A reply ends at the model's 0x04 or where the 2047-byte frame, prompt and reply together, is full.