OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Models

Hourglass Transformer Pytorch

Hourglass Transformer Pytorch is a Python/PyTorch library implementing a hierarchical transformer that shortens token sequences for intermediate layers and restores sequence length afterward. It includes language-model examples, including nested and non-autoregressive configurations, plus a token-based image example using linear projections for upsampling and downsampling.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Hourglass Transformer Pytorch — OpenViST project profile in AI Models.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use it when you want a modifiable PyTorch implementation for experimenting with hierarchical sequence processing, rather than a hosted model or a ready-made end-user application.

Who it is for

  • Researchers exploring hierarchical or sequence-length-reduced transformer architectures.
  • Developers prototyping autoregressive or non-autoregressive language-model configurations in PyTorch.
  • Developers experimenting with transformer processing of tokenized image inputs.

Documented uses

  • Configure a language model with sequence shortening and a chosen transformer depth.
  • Explore nested hourglass structures or a Funnel Transformer-like configuration.
  • Try non-autoregressive prediction using a non-causal setting and input mask.
  • Process tokenized image features with linear up/downsampling.
Before you choose

Requirements & limits

The implementation is available in a public GitHub repository under the MIT license; the README documents installation through pip.

  • The hierarchy is configured through shortening factors and a depth tuple, which can also be nested.
  • The language-model class returns position-wise logits in the README examples.
  • The image example uses HourglassTransformer with updown_sample_type='linear', rather than the language-model class.
  • The README identifies shift padding for naive downsampling as a TODO.
  • Non-autoregressive support is demonstrated, but the README says results were not presented in the paper.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Install the package and configure a language model

Set up the documented package and define an HourglassTransformerLM with a chosen sequence length and hierarchy.

Before you start

  • Python and PyTorch, as used in the README example.
  • A development environment in which the package can be installed with pip.

Steps

  1. Install the package with the command below.

    Documented command
    pip install hourglass-transformer-pytorch

  2. Import HourglassTransformerLM from hourglass_transformer_pytorch and import torch.

  3. Set model parameters such as num_tokens, dim, max_seq_len, shorten_factor and depth; the README example uses 256 tokens, dimension 512, maximum sequence length 1024, shortening factor 2 and depth (4, 2, 4).

  4. Supply a token tensor to the model; the README illustrates a (1, 1024) integer tensor and describes logits shaped (1, 1024, 256).

Expected outcome

A configured language-model instance and the documented pattern for obtaining per-position token logits.

Guide limitations

  • The README example uses randomly generated token IDs; it does not document data preparation or training this model configuration.
  • The repository notes a TODO for shift padding when using naive downsampling.
Documentation 1 ↗
Source-based practical guide

Configure a non-autoregressive language-model example

Adapt the README's model configuration for non-autoregressive sequence prediction.

Before you start

  • Python and PyTorch.
  • Token IDs and a mask shaped for the input sequence, following the README example.

Steps

  1. Create an HourglassTransformerLM with the chosen vocabulary size, dimensions, maximum sequence length, shortening factor and depth.

  2. Set causal=False, as shown in the README.

  3. Create the input token tensor and a boolean mask, then pass both to the model as model(x, mask=mask).

Expected outcome

The documented example returns logits for each input position, with output shape `(1, 1024, 20000)` for its stated parameters.

Guide limitations

  • The README says results for non-autoregressive use were not presented in the paper.
  • This is a configuration example; it does not specify a training or evaluation procedure.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
18 October 2026
Original website title
lucidrains/hourglass-transformer-pytorch · Official homepage title · retained in its source language
How we review sourcesSuggest a correction