OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Infrastructure & Serving

Fastgpt

Fastgpt is an open-source Python library that connects GPT models from Hugging Face Transformers to ONNX Runtime quantization for text generation, with a generation interface designed to work like Transformers' generate function.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Fastgpt — OpenViST project profile in AI Infrastructure & Serving.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use Fastgpt when you have a Transformers-based GPT text-generation workflow and want to investigate ONNX Runtime conversion and quantization through a compatible generation wrapper.

Who it is for

  • Python developers using Hugging Face Transformers who want to try ONNX Runtime-quantized GPT text generation.
  • Practitioners evaluating CPU inference performance for a particular model and configuration.

Documented uses

  • Load a documented GPT model through Fastgpt's ONNX generation wrapper.
  • Generate text from tokenized prompts using a Transformers-compatible generation interface.
  • Compare PyTorch and ONNX-based inference timings for a defined model and test setup.
Before you choose

Requirements & limits

The repository is public and identifies the project as MIT-licensed. The README documents installation through pip.

  • The README's quick example uses distilgpt2; model compatibility beyond documented cases should not be assumed.
  • The loading workflow includes model download, conversion, output comparisons and quantization, not only inference from a prebuilt ONNX file.
  • The project reports CPU timing comparisons for distilgpt2 on a two-core Intel Xeon E5-2673 v4; performance may differ with other hardware, models or generation settings.
  • The README states that CodeGen is not directly supported through the standard Transformers loading route and provides a separate example path.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Install Fastgpt and generate text with distilgpt2

Set up the documented Python example to generate text using Fastgpt's ONNX-backed Transformers-compatible model.

Before you start

  • Python environment with Fastgpt and the example's Transformers dependencies installed.
  • A model identifier supported by the documented example; the README uses distilgpt2.

Steps

  1. Install the package with the command below.

    Documented command
    pip install fastgpt

  2. Load CausalLMModelForOnnxGeneration from fastgpt using from_pretrained("distilgpt2").

  3. Load the matching tokenizer with AutoTokenizer.from_pretrained("distilgpt2") and tokenize a prompt.

  4. Pass the resulting input IDs to model.generate, then decode the returned IDs with the tokenizer.

Expected outcome

The example generates and prints decoded text from a prompt using Fastgpt's ONNX-backed generation model.

Guide limitations

  • The documented example uses distilgpt2; the README does not establish support for every Transformers GPT model.
  • The README says CodeGen models are not directly loadable through this standard path and directs users to a separate example.
Documentation 1 ↗
COMPARE OPTIONS

Compare options for Model inference and serving

These projects document a common task and share a project role. This is a source-based comparison, not an execution benchmark or a guarantee of interchangeability.

ProjectDocumented purposePlatformAccess and requirements
FastgptLoad a documented GPT model through Fastgpt's ONNX generation wrapper.CLIThe repository is public and identifies the project as MIT-licensed. The README documents installation through pip.
Cloud AI SDKAdapt and run documented model recipes for computer-vision and NLP inference.Cloud, ServerThe GitHub repository is public. Its public visibility does not establish license terms.
TurboquantReplace a model’s existing inference cache with a compressed cache.Cloud, Linux, macOS, Self-hostedThe project describes itself as open source. The README documents installation through pip or from a source checkout.
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
17 October 2026
Original website title
LowinLi/fastgpt · Official homepage title · retained in its source language
How we review sourcesSuggest a correction