A coding model of my own

This is a prototype, and I want to be plain about that. It was never going to compete with the frontier labs. It was built to answer one question, and it did.

What I did

I trained a language model from scratch, rather than fine-tuning someone else's, to write modern web code: Next.js, React, TypeScript and Tailwind.

The early versions were tiny. Each round got bigger, first in a simple GPT-style setup and then in a larger, more modern architecture. Pretraining taught it what web code looks like. Several rounds of instruction tuning then taught it to follow a request, using instruction-and-code pairs generated with the help of larger models.

All the heavy training ran on GPUs rented by the hour. Data preparation and evaluation ran on a laptop.

What it proved

One person can do this. No lab, no cluster, no team. That was the question, and the answer is yes.

What it cost to find out

The second lesson is less cheerful. A training run is not like a deploy. You cannot patch it afterwards.

Most individual runs are affordable. The danger is a wrong decision at the foundation, because fixing it means training the base model again, and at any useful size that is a bill in the thousands. One issue can cost you an entire iteration.

Data quality is necessary, not sufficient

I went in believing data quality was the whole game. It matters enormously. It is also one item on a longer list, and every item on that list cost me something:

• How you measure. I spent multiple rounds improving a score that did not reflect the job I actually wanted the model to do. Design the test around the real task before you train anything.
• The tokenizer. An early, convenient choice about how text is split into tokens turned out to be baked into everything downstream. It cannot be fixed without starting the base again.
• Training settings. A single wrong configuration value can crash a run or quietly ruin it hours in.
• Checkpoints and storage. Runs were lost to missing checkpoints, full disks and slow transfers, not to anything clever.
• The hardware you rent. Two machines with the same GPUs on paper can behave very differently in practice.
• Checking your own diagnosis. I stopped at least one healthy run because of a bug in the tool I was using to inspect it.

None of these are about the data, and any one of them can waste a run.

Where it stands

It is a working prototype that writes short, single-file components reasonably well and is much weaker at editing existing code and at full pages. It is not released, and I have no benchmark numbers I would stand behind publicly, because the tests were my own.

The next version would start from a different foundation. Whether I build it is a question of cost against curiosity, and so far curiosity has a good record.