OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Models

Marge Pytorch

Marge Pytorch is a Python/PyTorch library implementing Marge, a paraphrase-based pretraining method in which an encoder-decoder attention network learns to reconstruct a target document from a collection of evidence documents.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Marge Pytorch — OpenViST project profile in AI Models.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use it when you have tokenized document collections in memory-mapped form and want to experiment with Marge-style evidence-based reconstruction during pretraining.

Who it is for

  • Researchers exploring document-level pretraining objectives.
  • Developers training encoder-decoder models on collections of tokenized documents.

Documented uses

  • Train a model to reconstruct target documents from retrieved evidence documents.
  • Experiment with paraphrase-based pretraining as an alternative to masked language modeling.
  • Configure evidence and target documents from either the same collection or separate memory-mapped datasets.
  • Generate samples from the decoder conditioned on evidence documents.
Before you choose

Requirements & limits

The repository is public and specifies an MIT license. The README documents installation with pip.

  • The input preparation requirement is explicit: documents must be tokenized and stored as memory-mapped arrays.
  • Evidence retrieval is part of the training workflow, with a configurable number of evidence documents and periodic reindexing.
  • FAISS approximate-neighbor search can be disabled for collections where it is not needed.
  • The README marks attention distillation as experimental and disabled by default.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Set up a Marge training workflow

Install the package and prepare the documented model, data wrapper, and training loop.

Before you start

  • Python environment with PyTorch, NumPy, and the PyTorch DataLoader available.
  • Tokenized documents stored as memory-mapped arrays shaped by document count and sequence length; masks can be supplied in a corresponding memory-mapped array.
  • The documented example uses a vocabulary size, sequence length, model dimensions, and number of documents that must be adapted to the dataset.

Steps

  1. Install the package with the command below.

    Documented command
    pip install marge-pytorch

  2. Create a Marge model with parameters for vocabulary size, maximum sequence length, encoder and decoder depth, attention heads, and feed-forward dimensions.

  3. Pass the model and memory-mapped document and mask paths to TrainingWrapper, specifying document count, sequence length, and number of evidence documents.

  4. Create a PyTorch DataLoader from trainer.dataset; for each batch, call the trainer, backpropagate the loss, and perform the optimizer step.

  5. Call trainer.reindex() at suitable intervals, as in the README example, which reindexes every 10,000 steps.

Expected outcome

A configured training loop for learning to reconstruct target documents from retrieved evidence documents.

Guide limitations

  • The README does not specify a complete optimizer configuration or provide a real training corpus.
  • Input documents must already be tokenized and arranged in memory-mapped files.
  • Approximate-neighbor retrieval can be disabled with use_faiss_ann=False when it is not needed for a small document collection.
Documentation 1 ↗
Source-based practical guide

Configure separate evidence and target document sets

Point the training wrapper at distinct memory-mapped files for evidence documents and target documents.

Before you start

  • Evidence documents and target documents stored in separate memory-mapped arrays.
  • Target documents use the same sequence length as the evidence documents.
  • Optional corresponding memory-mapped mask files for each set.

Steps

  1. Set the wrapper’s document and mask paths to the evidence arrays.

  2. Pass num_targets and target_seq_len for the target collection.

  3. Set target_memmap_path and, if used, target_masks_memmap_path to the target arrays.

Expected outcome

A `TrainingWrapper` configured to draw evidence and target documents from separate sets.

Guide limitations

  • The README documents the wrapper configuration but does not provide a separate end-to-end training example for this setup.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
18 October 2026
Original website title
lucidrains/marge-pytorch · Official homepage title · retained in its source language
How we review sourcesSuggest a correction