OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Models

LLM Drop

LLM Drop is an open-source Python implementation for researching redundancy in transformer language models by dropping blocks, attention layers, MLP layers, or combinations of them. Its repository includes scripts for compression experiments, benchmark evaluation, speed and KV-cache measurement, and optional AWQ or GPTQ post-training quantization.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

LLM Drop — OpenViST project profile in AI Models.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use it when you need a code-based research workflow to investigate transformer-layer redundancy and evaluate the effects of removing model components.

Who it is for

  • Researchers studying transformer architectures and model compression
  • Practitioners comparing layer-dropping strategies for language models

Documented uses

  • Compare block, attention, MLP, and joint layer-dropping strategies.
  • Run provided dropping pipelines on transformer models.
  • Evaluate dropped checkpoints with language and reasoning benchmarks.
  • Measure inference speed and KV-cache changes.
  • Apply optional 4-bit AWQ or GPTQ quantization to dropped models.
Before you choose

Requirements & limits

The public GitHub repository specifies an Apache-2.0 license. The documented setup uses Python 3.10 and Conda.

  • The project provides distinct block, attention, MLP, and joint dropping pipelines, allowing different compression approaches to be studied.
  • The documented workflow includes task benchmarking and separate scripts for speed and KV-cache measurements.
  • FlashAttention is listed in the core installation steps; AutoAWQ and AutoGPTQ are listed for optional quantization.
  • Reported speed and benchmark results are specific to the configurations shown in the repository and should not be assumed for other models or environments.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Set up the LLM Drop research environment

Install the repository and its core documented dependencies in a Python 3.10 Conda environment.

Before you start

  • Conda
  • Python 3.10
  • Git
  • FlashAttention installation prerequisites

Steps

  1. Create and activate a Conda environment named llm-drop with Python 3.10.

  2. Clone the LLM-Drop repository and change into its directory.

  3. Install the project in editable mode with pip install -e ., then install FlashAttention with pip install flash-attn --no-build-isolation.

Expected outcome

The repository and core dependencies are installed in the named Conda environment.

Guide limitations

  • This covers setup only; it does not establish that a dropping run or benchmark will succeed in a particular environment.
  • The README lists additional AutoAWQ and AutoGPTQ installation steps only for optional quantization.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
13 October 2026
Original website title
CASE-Lab-UMD/LLM-Drop · Official homepage title · retained in its source language
How we review sourcesSuggest a correction