I Trained a 757M-Parameter LLM From Scratch, Then Fine-Tuned Qwen Instead. Here's Everything I Learned
Most "I built an LLM" posts end at the loss curve. The curve goes down, there's a screenshot of a terminal, and everyone claps.
The interesting post is the one where the from-scratch model loses — and the author tells you why.
That's what happened here. Two language models, one task (empathetic, supportive dialogue), built head to head: a GPT-style transformer trained from random init at 253M and 757M parameters in PyTorch on consumer hardware — and Qwen2.5-0.5B-Instruct fine-tuned with QLoRA in two stages, merged back to a single fp16 checkpoint, wrapped in a FastAPI serving layer with OpenAI-compatible endpoints, Docker, and Prometheus metrics.
The fine-tune won. Not narrowly. It won on the axis that actually matters, which is "does this thing behave like a helpful assistant," and it won for reasons that are structural, not skill-based.
If you're planning to burn GPU-weeks on a from-scratch run, read this before you rent the box.

图示:来源页预览图(仅作编辑上下文;版权归属原发布方) · 来源引用:dev.to
The from-scratch model didn't fail at language modeling
This is the part people misread. A 757M model written by hand and trained on a decent corpus does learn English. It produces grammatical sentences. The loss curve behaves. Perplexity drops. If your benchmark is "does the output look like text," you'd call it a success.
Then you ask it to respond empathetically to someone who just lost their job, and it writes a plausible-sounding continuation of a story about someone who just lost their job. It doesn't answer. It continues. It has learned the shape of dialogue without learning the contract of dialogue, because the contract lives in data you don't have and post-training you didn't do.
The fine-tuned Qwen answered. It asked a follow-up question. It didn't hallucinate a therapist's credentials. It stayed in register.
That gap is not a parameter-count gap. It's a token-budget and post-training gap, and it's enormous.
Three reasons the from-scratch model lost
1. You cannot out-token a frontier lab on a consumer budget.
Chinchilla-style scaling puts compute-optimal training at roughly 20 tokens per parameter. For a 757M model, that's on the order of 15 billion tokens. At realistic single-machine throughput, that's not a weekend — it's weeks, and that's before you've done a single hyperparameter sweep. Anyone who "trained a 757M model from scratch" on hobby hardware either trained it on far fewer tokens than optimal (which is fine, but then it's undertrained) or spent a very long time.
Qwen2.5-0.5B already absorbed trillions of tokens of curated, deduplicated, filtered, multilingual, code-and-math-inflected data. You are not going to replicate that data pipeline in a few weeks. You are not going to replicate it in a few years, solo.
2. Instruction-following is a post-training problem, not a pre-training problem.
The from-scratch run is pure next-token prediction. Qwen2.5-0.5B-Instruct has already been through SFT and preference optimization. When you then QLoRA it, you're not teaching it to talk — you're teaching it a style and a task. That's a vastly smaller delta, which is why LoRA ranks of 16-64 work at all. You're nudging a well-conditioned model, not bootstrapping a personality from noise.
3. The eval is unfair, and that's the point.
The from-scratch model was judged on instruction-following and answer quality — exactly the axes post-training optimizes. It never had a chance. But that's not a rigged comparison; it's the actual production requirement. Nobody ships a chatbot because its perplexity was good.
The counter-argument here is real, though, and it deserves airtime.
Counterpoint: from-scratch still buys you things fine-tuning can't
Fine-tuning a base model is renting. You inherit the tokenizer, the license, the safety behavior, the refusal boundaries, the multilingual coverage, and the blind spots. For most teams that's the correct trade. For some, it isn't:
- License and provenance control. If your product can't carry an upstream model license, fine-tuning is a non-starter regardless of quality.
- Domain-shifted tokenizers. If your corpus is genomic sequences, log formats, or a low-resource language with a hostile tokenizer, a from-scratch model with a bespoke tokenizer can be dramatically cheaper at inference for the same capability.
- Research and teaching. The 757M run taught the author more about transformers than any paper. That has value that doesn't show up in a win/loss column.
- Distillation targets and privacy. Training a small model on data that can never leave your VPC is a legitimate reason to own the whole stack.
None of these are "my from-scratch model is better at chat." They're "I need to own the artifact."
The actual lesson: your moat is not the base model
Here's the trend take. In 2023, "we trained our own model" was a flex. In 2026, it's usually a misallocation of engineering time. The frontier moved; a half-billion-parameter instruct model that fits in a Docker container now handles a startling amount of narrow, well-scoped work — and the remaining delta is closed with data, retrieval, evals, and serving, not with gradient updates you ran yourself.
What actually determined quality in this experiment:
- The dataset. Curated supportive-dialogue examples beat raw volume.
- The two-stage schedule. Broad first, narrow second, merge to fp16 for serving.
- The harness. OpenAI-compatible endpoints meant the same eval and client code could hit both models unchanged. Prometheus meant you could see latency and token counts, not just vibes.
Notice that none of those are "the number of parameters." The 253M and 757M from-scratch models were separated by 3x compute and a rounding error of difference in usefulness. The fine-tune separated itself immediately.

图示:HEXOPS 原创配图(受选题启发,非复制原文图片) · 来源引用:dev.to
Your 30-day action plan
Week 1 — Define the task and the eval before you touch a GPU. Write 50 held-out prompts with rubric-scored expected behaviors. Pick your baseline: an off-the-shelf instruct model. Measure it. You now have a number to beat. If you can't write the eval, you don't have a task yet.
Week 2 — Build the serving layer first, model second.
FastAPI, OpenAI-compatible /v1/chat/completions, Docker, request logging. Point it at the baseline model. This sounds backwards. It isn't — it's what makes every later comparison possible, and it's the step hobbyists skip.
Week 3 — Fine-tune with QLoRA on a small instruct model. Start with 0.5B-3B. 4-bit NF4 quantization, LoRA rank 16-32, 1-3 epochs, a few thousand curated examples. Run stage one broad, stage two narrow. Version your dataset. Log hyperparameters. Compare against Week 1's rubric, not against a loss curve.
Week 4 — Merge, deploy, and decide. Merge adapters into a single fp16 checkpoint for clean serving. Re-run the eval. If you beat baseline, ship it. If you don't, the problem is your data — iterate on examples, not architecture. Only if you're still stuck after three data iterations should you consider touching model internals — and even then, fine-tune a bigger base before you consider training from scratch.
Budget check: Week 3 costs single-digit dollars on a rented A10 or a few hours on a consumer 3090/4090. The from-scratch run costs weeks and teaches you less about shipping.
Bottom line
Train a model from scratch once, for the education. It's worth it — the intuition you get from watching your own 757M model fail to answer a question is not something a paper gives you.
Then fine-tune someone else's model for the product. The base model isn't your moat. Your data, your eval, and your serving layer are. Every hour you spend re-deriving a tokenizer is an hour you didn't spend on the three things that actually move quality.
The loss curve going down is the least interesting part of the story. The interesting part is knowing when to stop watching it.
Sources
- Original write-up: I Trained a 757M-Parameter LLM From Scratch, Then Fine-Tuned Qwen Instead
- Hoffmann et al., Training Compute-Optimal Large Language Models (Chinchilla scaling laws): https://arxiv.org/abs/2203.15556
- Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs: https://arxiv.org/abs/2305.14314
- Hu et al., LoRA: Low-Rank Adaptation of Large Language Models: https://arxiv.org/abs/2106.09685
- Qwen2.5-0.5B-Instruct model card: https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct
- FastAPI: https://fastapi.tiangolo.com/ · Prometheus: https://prometheus.io/



