OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Developer Tools & SDKs

Hyper Connections

Hyper Connections is an MIT-licensed Python library that implements multiple residual streams for integration into neural-network models, with documented PyTorch helpers for wrapping branch layers and expanding and reducing streams.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Hyper Connections — OpenViST project profile in AI Developer Tools & SDKs.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use it when adapting a neural-network model to experiment with multiple residual streams or the documented fractionated feature-dimension option.

Who it is for

  • PyTorch developers experimenting with residual-stream architecture changes.
  • Researchers and model builders prototyping neural-network layers based on Hyper-Connections.

Documented uses

  • Add multiple residual streams to a PyTorch model.
  • Wrap a branch layer so it can operate within an expanded residual representation.
  • Compare the helper-based pattern with plain residual behavior using the documented disable option.
  • Explore fractionated feature dimensions through the documented `num_fracs` parameter.
Before you choose

Requirements & limits

The repository is public and specifies an MIT license. The README documents installation with pip.

  • The documented workflow requires deciding the number of streams and matching the configured feature dimension to the model branch.
  • The README identifies stream-reduction placement relative to final transformer normalization as an unresolved implementation choice.
  • The README includes a disable=True option for comparing the helper pattern with plain residual behavior.
  • Fractionated feature dimensions are exposed through the num_fracs parameter.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Integrate four residual streams around a branch layer

Adapt a PyTorch branch layer using the README's documented expand, wrap and reduce pattern.

Before you start

  • Python environment with PyTorch installed
  • Install the library with pip install hyper-connections
  • A branch layer and residual tensor whose feature dimension matches the configured dimension

Steps

  1. Import PyTorch and get_init_and_expand_reduce_stream_functions from hyper_connections.

  2. Fetch the initialization, expansion and reduction helpers with a stream count of 4.

  3. Wrap the branch with init_hyper_conn(dim=512, branch=branch) when using the README's 512-feature example.

  4. Expand the residual before the trunk, pass it through the wrapped branch, then reduce the streams by summation after the trunk.

Expected outcome

A model branch integrated into the documented four-stream residual pattern.

Guide limitations

  • The four-stream and 512-feature values are example settings, not universal model requirements.
  • The README says it is uncertain whether a transformer should reduce streams before or after its final normalization.
Documentation 1 ↗
Source-based practical guide

Set up the manual branch-input and residual-add pattern

Use the README's manual pattern to separate branch input from the residual-add operation.

Before you start

  • Python environment with PyTorch installed
  • Install the library with pip install hyper-connections
  • A branch layer and residual tensor with a feature dimension matching the configured dimension

Steps

  1. Fetch the helper functions for the desired stream count and initialize a Hyper Connection with the matching feature dimension.

  2. Expand the residual tensor into the selected number of streams.

  3. Pass the expanded tensor to the Hyper Connection to obtain branch_input and add_residual.

  4. Run the branch on branch_input, pass its output to add_residual, and reduce the streams after the trunk.

Expected outcome

A branch integrated using distinct branch-input and residual-add functions, followed by stream reduction.

Guide limitations

  • The README's manual example uses four streams and a 512-feature branch.
  • For transformer models, the README leaves the placement of reduction relative to final normalization unresolved.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
8 October 2026
Original website title
lucidrains/hyper-connections · Official homepage title · retained in its source language
How we review sourcesSuggest a correction