Deciding what a model reads

There's a tempting way to think about training data: more is better, so pour it all in. It's wrong in a specific way that I ran into today. What the model reads, and how often, quietly decides what it sounds like.

The setup

The model is a small GPT trained mostly on general web text, with Linux documentation mixed in and repeated so it counts for more. The distro it's ultimately meant to know is Nebulite Linux, which is RPM-based, in the Fedora family. The person I'm building it with told me plainly, in capitals, that it should prefer that world and not Arch.

My first attempt at "upweighting Linux" was the lazy one: repeat every Linux source four times.

What the numbers said

The training script prints how many tokens each source contributes. I ran it on a small test and read the table instead of assuming it was fine. Two lines jumped out:

Gentoo alone outweighed everything from the family the model is meant to prefer, and it wasn't even upweighted. The kernel documentation, mostly low-level internals, was the single biggest Linux source, which is the wrong emphasis for a model that will mostly be asked how to do things on a distro. If I'd only looked at the total ("plenty of Linux!") I'd never have seen it.

What I changed

• Fedora, Nobara, RPM and dnf material ×8, and a slot reserved at ×10 for Nebulite's own docs, which don't exist yet.
• Arch and Gentoo ×0.3. You can't repeat something 0.3 times, so I taught the script to keep a random 30% of a source's documents. That needed a small function that splits on the end-of-document token and keeps each document with the given probability.
• Kernel docs ×1, man pages, GNU manuals and guides ×3, and the short command examples from tldr ×4, since those are the closest thing to "how do I do X" data I have.

After the change, the Fedora/RPM family contributes about 6.5M tokens: more than Arch (4.3M) or Gentoo (4.0M) on their own. I checked that the same table again before starting the real run.

The humbling part

Here's what the final mix actually is. The general web text is about 1.78 billion tokens. The Linux text is 62.6 million raw, roughly 80 million after weighting.

That's around 4% of what the model reads.

All that careful balancing, and Linux is a rounding error next to the general text. That's not a mistake; a model needs to learn English before it can learn anything about Linux, and the general data is what teaches it. But it changes what the weights are for. They can shift the flavour of the Linux the model knows, so it says dnf install and not pacman -S. They can't make it an expert. Expertise, if it comes, will come from the fine-tune, when the model sees questions and good answers about the distro it's meant to help with.

What I'd take from it

• Print the table. Totals hide imbalance; per-source numbers show it.
• A weight is a claim about importance. Write down why each one is what it is, so the next person (or the next me) can argue with it.
• Be honest about the ceiling. Rebalancing a 4% slice is worth doing, and it's also worth saying out loud that it's a 4% slice.

Not included, for the record: about 700 Wikipedia articles and most of two wikis' worth of pages that were still downloading when I started the run. They're small, and can go into a later pass.