OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Infrastructure & Serving

Swiftinfer

SwiftInfer is a Python project that implements Streaming-LLM with TensorRT-LLM for large language model inference and serving. Its README provides a Llama workflow for building a TensorRT engine and running multi-round conversations, and describes benchmark comparison with the original PyTorch implementation.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Swiftinfer — OpenViST project profile in AI Infrastructure & Serving.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use SwiftInfer when you want a documented TensorRT-LLM-based Llama workflow for streaming inference, or a starting point for the README's Streaming-LLM benchmark comparison.

Who it is for

  • Developers building self-hosted LLM inference or serving workflows on TensorRT-LLM.
  • Practitioners exploring Streaming-LLM behavior with Llama models.
  • Teams setting up an inference benchmark against the original PyTorch Streaming-LLM implementation.

Documented uses

  • Build a TensorRT engine from a local Llama-family model directory.
  • Run a multi-round conversation example using MT-Bench question data.
  • Compare SwiftInfer's Streaming-LLM implementation with the original PyTorch version.
Before you choose

Requirements & limits

The repository is public and lists an Apache-2.0 license. The documented workflow is self-hosted and requires local software dependencies and model files.

  • The README pins the implementation to TensorRT-LLM v0.6.0, citing rapidly changing APIs.
  • The non-Docker setup lists CUDA, cuDNN, NCCL, TensorRT and PyTorch, with CUDA toolkit 12.2 or later and TensorRT 9.1.0 or later.
  • The Llama example requires a local model directory and separately downloaded MT-Bench question data.
  • The README describes a benchmark against the PyTorch implementation and gives its test hardware, but those details do not define universal requirements.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Build and run the documented Llama example

Build a TensorRT engine for a local Llama model and run the example conversation workflow.

Before you start

  • A SwiftInfer checkout and TensorRT-LLM v0.6.0.
  • A local Hugging Face model directory for meta-llama/Llama-2-7b-chat-hf or another Llama-based variant.
  • MT-Bench question data from LMSYS-FastChat.
  • For the non-Docker installation route: Python, build essentials (gcc/g++, make, cmake), CUDA toolkit, cuDNN, NCCL, TensorRT, and PyTorch; TensorRT 9.1.0 or later and CUDA toolkit 12.2 or later.

Steps

  1. Install TensorRT-LLM v0.6.0, using the README's Docker route or its listed non-Docker prerequisites and installation instructions.

  2. Clone SwiftInfer and install it with pip install .; for the non-Docker route, provide TRT_ROOT, NCCL_ROOT and CUDNN_ROOT as shown in the README.

  3. From examples/llama, run build.py with --model_dir set to the local model path to build the TensorRT engine.

  4. Download the MT-Bench question.jsonl file to mt_bench_data, then run ../run_conversation.py with the tokenizer directory, engine directory and input file paths.

  5. Keep --only_n_first 5 to evaluate the sample's first five questions, or remove the argument to evaluate all samples.

Expected outcome

The documented Llama example runs a multi-round conversation using the built TensorRT engine and MT-Bench questions.

Guide limitations

  • The README ties SwiftInfer to TensorRT-LLM v0.6.0 because its API was changing rapidly; it does not document compatibility with later versions.
  • Model files and benchmark question data must be obtained separately.
  • The commands and sample settings describe the repository workflow; they are not evidence of an independently executed test.
Documentation 1 ↗
COMPARE OPTIONS

Compare options for Model inference and serving

These projects document a common task and share a project role. This is a source-based comparison, not an execution benchmark or a guarantee of interchangeability.

ProjectDocumented purposePlatformAccess and requirements
SwiftinferBuild a TensorRT engine from a local Llama-family model directory.CLI, Linux, Self-hostedThe repository is public and lists an Apache-2.0 license. The documented workflow is self-hosted and requires local software dependencies and model files.
MorphingdbImport a libtorch model and create predictions from values in a PostgreSQL table.Docker, Linux, Self-hosted, ServerThe README documents a Docker image and a build-from-repository option, followed by connecting to PostgreSQL and enabling the pgdl extension.
Gateway — pyadavSend chat-completion requests to a provider through a common server endpoint.ServerThe repository is public and identifies the project as Apache-2.0 licensed. README installation instructions include binaries, platform package managers and Docker.
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
15 October 2026
Original website title
hpcaitech/SwiftInfer · Official homepage title · retained in its source language
How we review sourcesSuggest a correction