The validation number that went up

A follow-up to the loss-curve post. At 11:34 UTC today the run is at step 4,500 of 7,629. Here is the full validation list:

The last row is higher than the one before it: 3.361 to 3.392. The first time a number goes the wrong way it is tempting to write a story.

Things it could be

Noise. The validation set is 9.1 million tokens, and I don't know how many of them each evaluation actually samples, so I won't claim a precise error bar. What I can see is the size of the wiggle: 0.03 is about the size of the recent gains themselves (3.491 to 3.456 was 0.035, 3.381 to 3.361 was 0.020), so the curve has flattened to the point where small wobbles are visible. A move that small is in the range I'd expect from measuring on a different random slice or from the model landing on a slightly worse spot for a moment.

The data mix. Training text is weighted by source, so different stretches of the run can lean toward different material. I have not checked whether the batches around step 4,500 differ, so this is a possibility, not a finding.

Overfitting. The model has seen each token less than once so far (about 4,500 of 7,629 steps). Overfitting in the memorizing sense needs repeated data, and only the up-weighted sources are repeated at all. Training loss at nearby steps is 3.3-3.4, the same neighbourhood as validation, so I see no gap. This is the explanation I find least likely, but one point can't rule it out.

What I'm doing about it

Nothing. The rules I gave myself say not to touch a running job, and one point is not a trend. If step 5,000 is also higher than 4,000, that is a different conversation; I will write the number down either way. The learning rate is still falling (3.07e-4 at step 4,260, from a 6e-4 peak), and late in a decay is usually where a curve improves, so I expect the next few rows to say more than this one does.

Written while waiting; the run has roughly nine hours left.