OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Developer Tools & SDKs

Beaver

Beaver is an open-source, modular Python framework and reproducible code pipeline for large language model alignment research. It supports supervised fine-tuning (SFT), RLHF and Safe RLHF, including preference-model, reward-model and cost-model training, with associated training data and evaluation workflows.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Beaver — OpenViST project profile in AI Developer Tools & SDKs.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use Beaver when you need an open-source code pipeline for training and researching LLM alignment methods, especially experiments that connect SFT, preference learning and safety-constrained RLHF.

Who it is for

  • Researchers studying LLM alignment and constrained alignment.
  • ML engineers building or adapting SFT, RLHF and Safe RLHF training experiments.
  • Teams working with supported pretrained model families such as LLaMA, OPT and Baichuan.

Documented uses

  • Fine-tune a supported pretrained model with supervised data.
  • Train preference, reward and cost models for alignment experiments.
  • Run RLHF or Safe RLHF training to study reward optimization with safety-cost constraints.
  • Adapt datasets and training parameters for experiments.
  • Evaluate and compare trained models using the documented evaluation workflows and interactive arena.
Before you choose

Requirements & limits

The repository identifies Beaver as open source and lists the Apache-2.0 license. Its documented use involves cloning and running the Python project; the training workflow uses a Weights & Biases API key.

  • The repository supports SFT, RLHF and Safe RLHF for model families including LLaMA, OPT and Baichuan.
  • The project describes human-labeled preference data of up to one million pairs and support for customized datasets.
  • Training can be compute-intensive: the README’s listed training processes were tested with LLaMA-7B on eight NVIDIA A800-80GB GPUs.
  • Documented options include DeepSpeed ZeRO-Offload for reducing peak GPU memory use and multi-node training.
  • The framework includes reward and cost model training, with evaluation examples that include BIG-bench and GPT-4 Evaluation.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Set up Beaver’s training environment

Prepare the repository and dependencies using the documented native Conda or Mamba setup.

Before you start

  • A machine with Git and Conda or Mamba available.
  • Access to the safe-rlhf GitHub repository.

Steps

  1. Clone https://github.com/PKU-Alignment/safe-rlhf.git and change into the repository directory.

  2. Create the environment with conda env create --file conda-recipe.yaml or mamba env create --file conda-recipe.yaml.

  3. Activate the environment with conda activate safe-rlhf.

Expected outcome

The repository’s documented Conda environment is created and activated.

Guide limitations

  • This guide covers environment setup only, not a completed training run.
  • The repository also documents a container route, which requires NVIDIA Container Toolkit and NVIDIA Docker setup.
Documentation 1 ↗
Source-based practical guide

Run the documented SFT-to-Safe-RLHF training stages

Use Beaver’s documented scripts to fine-tune a model, train reward and cost models, and proceed through RLHF stages.

Before you start

  • A configured Beaver environment or the documented Docker runner.
  • A supported pretrained model or checkpoint and sufficient compute for the chosen model and settings.
  • A Weights & Biases API key, exported as WANDB_API_KEY.
  • For the README’s listed LLaMA-7B training processes, the stated test setup used eight NVIDIA A800-80GB GPUs; the repository also describes ZeRO-Offload and multi-node options.

Steps

  1. Activate the safe-rlhf environment or start the documented Docker runner, then export WANDB_API_KEY.

  2. Run the command below.

    Documented command
    bash scripts/sft.sh --model_name_or_path --output_dir output/sft

  3. Train value models with the command below and the documented cost-model script using output/sft as its model path.

    Documented command
    bash scripts/reward-model.sh --model_name_or_path output/sft --output_dir output/rm

  4. Continue with Beaver’s documented RLHF and Safe RLHF training scripts, adjusting script parameters such as GPU count and batch size for the machine.

Expected outcome

The documented workflow progresses from SFT to reward and cost model training, followed by RLHF alignment stages and their output checkpoints.

Guide limitations

  • The README notes that training parameters may need adjustment for the machine setup.
  • The supplied training instructions are partial for later stages; consult the repository README for the complete commands and options.
  • ZeRO-Offload can alleviate peak GPU memory usage but does not remove the need to provision compute.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
11 October 2026
Original website title
PKU-Alignment/safe-rlhf · Official homepage title · retained in its source language
How we review sourcesSuggest a correction