LLMs From Scratch: Build GPT-like Models in Notebooks

Understand the project

Practical overview

Use it when you want a code-led learning path into LLM construction and architecture, rather than a hosted model or API.

Learners with basic Python who want to understand language models by implementing their components.
Developers and researchers exploring model architecture through notebook-based lessons.

Study text processing and attention as foundations for language models.
Follow code and notebooks to implement a GPT-like model and work through pretraining concepts.
Explore architecture discussions and notebooks for models such as ChatGLM3, Llama 3 and RWKV6.

Before you choose

The repository is public. Its README badge identifies the code license as Apache 2.0.

The materials are Jupyter Notebooks, and the project description names basic Python as the starting prerequisite.
The README recommends Codes notebooks for a quicker introduction and Translated_Book notebooks for more detailed study.
The chapter plan lists fine-tuning and practical-use chapters as forthcoming, while the foundational materials cover encoding, pretraining and fine-tuning code.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Choose a quick-start or detailed notebook path for studying the fundamentals of building a GPT-like language model.

Basic Python knowledge; the project describes the learning path as suitable for learners with Python basics.
For the notebook code, the README notes PyTorch basics.

Open the repository and choose the Codes directory for a quick introduction or Translated_Book for more detailed study.
For the quick-start path, begin with a notebook listed in Codes; the README identifies chapter notebooks on text processing, attention, GPT implementation, and pretraining.
Continue through the notebooks to study the documented progression from text processing and attention to GPT construction and pretraining.

A structured notebook-based introduction to GPT-like model components and the documented coding stages of encoding, pretraining, and fine-tuning.

The materials are presented primarily for education and for developing small functional models, not as evidence of building a ChatGPT-scale foundation model.
The README lists chapters on text-classification fine-tuning, human-feedback fine-tuning, and practical LLM use as forthcoming.
The README does not specify environment setup or a verified execution procedure for the notebooks.

Evidence and transparency

Source record details

Source availability: Active
Sources checked: 15 October 2026
Original website title: datawhalechina/llms-from-scratch-cn · Official homepage title · retained in its source language