Practical overview
When it fits
Use it as a learning resource when you want to follow an LLM pipeline in code, from data preparation through generation and post-training. The project describes a CPU path for small models and GPU requirements for larger experiments.
Who it is for
- Students learning how transformer language models are trained.
- Developers who want to read or adapt the project’s scripts.
- Researchers exploring small-model post-training methods.
Documented uses
- Learn how raw text is prepared and converted into tokens for training.
- Study transformer components and next-token pretraining.
- Train a small model and generate text.
- Explore supervised fine-tuning, reward modeling and preference or reinforcement-learning methods.
Requirements & limits
The GitHub repository is public and lists an MIT license.
- The tutorial expects basic object-oriented programming, neural-network and PyTorch knowledge.
- A CPU laptop track is described for small models; larger training runs require a GPU.
- The repository presents both an original transformer and a more modern model path.
- The project demonstrates a small model’s output, which may be incoherent; training a model does not itself establish fluent performance.
Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.
Set up the project and install its core package
Clone the repository and install the project in editable mode.
Before you start
- Python 3.10 or later, as indicated by the repository badge.
- Git and pip.
Steps
Clone the repository with the command below.
git clone https://github.com/FareedKhan-dev/train-llm-from-scratch.gitChange into the project directory with
cd train-llm-from-scratch.Install the project in editable mode with the command below.
pip install -e .
Expected outcome
The project is installed in editable mode, which puts `config`, `src`, `data_loader`, and `ui` on the import path.
Guide limitations
- On Linux without an NVIDIA GPU, the README says to install the CPU build of PyTorch before the project, using
pip install torch --index-url https://download.pytorch.org/whl/cpu. - Optional training, UI, and documentation dependencies are installed separately through the listed extras; the core install alone does not install them.
Train a small language model on a laptop CPU and generate text
Prepare TinyStories data, train the documented tiny model on CPU, and generate a story.
Before you start
- A project environment with Python and dependencies installed.
- A laptop CPU; the README presents this track for use without a GPU.
Steps
Run the command below to download TinyStories and train a BPE tokenizer from scratch.
python scripts/prepare_tiny_data.pyRun the command below to train the 369K-parameter model.
python scripts/train_transformer.py --preset tiny --arch modernRun the command below to write a story.
python scripts/generate_text.py --model_path models/tiny.pt
Expected outcome
The documented pipeline prepares data, trains a small model, and writes generated text.
Guide limitations
- The README estimates about five minutes for training and about ten minutes for the complete laptop track; these are estimates, not guaranteed timings.
- The data-preparation command downloads 25 MB of TinyStories.
- The recipe covers the tiny laptop model, not larger models.
Sources & review
Source record details
- Source availability
- Active
- Sources checked
- 7 October 2026
- Original website title
- FareedKhan-dev/train-llm-from-scratch · Official homepage title · retained in its source language