OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Infrastructure & Serving

Distributed Llama

Distributed Llama is C++ infrastructure for distributing large language model inference across connected devices. Its root node loads model weights, synchronizes network state and processes part of the model, while worker nodes process other parts. It uses tensor parallelism and Ethernet synchronization, and offers command-line inference and chat, worker, and API-server commands.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Distributed Llama — OpenViST project profile in AI Infrastructure & Serving.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use it when you can connect a supported set of devices over Ethernet and want inference workloads and model memory divided across a cluster.

Who it is for

  • People with multiple connected devices who want to distribute LLM inference across them.
  • Developers building or studying local multi-device inference setups.

Documented uses

  • Distribute model inference across a root device and worker devices.
  • Run inference or chat from the command line.
  • Start worker nodes or an API server.
Before you choose

Requirements & limits

  • The project documents Linux, macOS and Windows support, with optimization for ARM and x86_64 AVX2 CPUs.
  • A cluster must use a power-of-two node count, and its maximum size depends on the model's number of KV heads.
  • The root node requires somewhat more RAM than worker nodes; model memory is split across the nodes.
  • The README lists Python 3 and a C++ compiler as prerequisites for its one-command root setup.
  • The repository identifies its license as MIT.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Prepare a root node with a listed model

Set up the root node and download a model and tokenizer using a documented launcher command.

Before you start

  • Python 3
  • A C++ compiler
  • A model listed in the README

Steps

  1. Choose a model from the README's launcher table.

  2. Run the corresponding the command below command shown for that model.

    Documented command
    python launch.py

Expected outcome

The launcher command is documented to download the selected model and tokenizer for root-node setup.

Guide limitations

  • This setup stage does not cover configuring or starting worker nodes.
  • Cluster node counts must be 1, 2, 4 or another power of two, and the maximum depends on the model's number of KV heads.
  • Supported quantization pairings are q40 models with q80 buffer-float-type, or f32 models with f32 buffer-float-type.
Documentation 1 ↗
COMPARE OPTIONS

Compare options for Model inference and serving

These projects document a common task and share a project role. This is a source-based comparison, not an execution benchmark or a guarantee of interchangeability.

ProjectDocumented purposePlatformAccess and requirements
Distributed LlamaDistribute model inference across a root device and worker devices.
MorphingdbImport a libtorch model and create predictions from values in a PostgreSQL table.Docker, Linux, Self-hosted, ServerThe README documents a Docker image and a build-from-repository option, followed by connecting to PostgreSQL and enabling the pgdl extension.
Gateway — pyadavSend chat-completion requests to a provider through a common server endpoint.ServerThe repository is public and identifies the project as Apache-2.0 licensed. README installation instructions include binaries, platform package managers and Docker.
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
10 October 2026
Original website title
b4rtaz/distributed-llama · Official homepage title · retained in its source language
How we review sourcesSuggest a correction