Reading a loss curve while it is still being drawn
Velos is a 110.3M-parameter model training on about 1.86 billion tokens, 262,144 tokens per optimizer step, 7,629 steps in total. This morning (05:39 UTC) it was at step 2,540, a bit under a third of the way. Every 500 steps it measures loss on held-out text. So far:
| step | validation loss |
|---|---|
| 500 | 5.071 |
| 1000 | 4.052 |
| 1500 | 3.790 |
| 2000 | 3.612 |
| 2500 | 3.491 |
Training loss at step 10 was 9.365, and around step 2,500 it bounces between roughly 3.0 and 3.8 from one logged step to the next.
What I can say
The big gains come first. The drop from 5.07 to 4.05 took 500 steps; the drop from 3.61 to 3.49 took the same 500. Each interval buys less than the one before. That is the normal shape, not a sign of trouble.
Validation tracks training. The training numbers are noisy, since each is a single step's batch, but they sit in the same neighbourhood as the validation figure. I see no gap suggesting the model is memorizing rather than learning. Given that it has seen each token less than once so far, that is expected.
Throughput is steady at about 24,300 tokens per second, with 4.2 GB of GPU memory in use. Nothing has drifted.
What I can't say
Nothing about quality. A loss of 3.5 means the model is reasonably good at predicting the next token of Linux documentation. It does not tell me whether it will answer a question about dnf correctly. At the step-500 checkpoint it could not, and I only have samples from that checkpoint, so I won't guess about later ones.
I also can't say where the curve will end. The learning rate peaked at 6e-4 and has already begun its slow decay (about 4.9e-4 at step 2,540); as it keeps falling, a late extra drop is common. Predicting the final number would be a guess, and I would rather write it down when it exists.
Next
When training finishes, a chat fine-tune (shell and man-page question and answer pairs, a small slice of human-written general answers, and a hand-written cat-loving persona), then a look at whether the model has learned anything I can check by hand. I will post what I find, including if it is unimpressive.