OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Infrastructure & Serving

Inferllm

InferLLM is a C++ framework for local inference with supported quantized large language models, offering CPU execution, optional CUDA GPU support, and an Android cross-compilation path.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Inferllm — OpenViST project profile in AI Infrastructure & Serving.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use InferLLM when you need a C++ local inference framework and your model is among its documented supported families. Choose the CPU path by default, or consider CUDA or Android only when those documented build paths match your target.

Who it is for

  • Developers building local inference tools for supported quantized language models.
  • Teams exploring CPU inference or CUDA-enabled inference on supported systems.
  • Developers adapting command-line inference to Android devices through the documented NDK and adb workflow.

Documented uses

  • Compile a local command-line runtime for a supported quantized model.
  • Run documented ChatGLM inference on a host or through adb on an Android device.
  • Build with the optional CUDA path when targeting supported CUDA systems.
  • Use model-specific ChatGLM or Baichuan applications with the framework.
Before you choose

Requirements & limits

The public repository states that InferLLM is licensed under Apache License 2.0.

  • GPU support is disabled by default; enabling it uses the documented ENABLE_GPU build option.
  • The README says CUDA is currently the only supported GPU option and requires the CUDA toolkit.
  • Android builds require the NDK and NDK_ROOT configuration.
  • Supported models have application-specific instructions; confirm the model format and application path before building around one.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Build and run the documented ChatGLM command-line example

Compile InferLLM locally and use the documented ChatGLM invocation with a model file.

Before you start

  • A supported ChatGLM model file, such as the documented chatglm-q4.bin example.
  • CMake and make for local compilation.

Steps

  1. Create a build directory, enter it, and run cmake .. followed by make.

  2. Follow the ChatGLM application documentation for the model-specific application.

  3. Run ./chatglm -m chatglm-q4.bin -t 4 as the documented local example.

Expected outcome

A local ChatGLM inference invocation using the CPU by default.

Guide limitations

  • The example depends on the named model file and ChatGLM application.
  • The README's four-thread recommendation refers to its x86 profiling result.
Documentation 1 ↗
Source-based practical guide

Cross-compile the documented command-line path for Android

Build InferLLM for Android and use its documented command-line inference application through adb.

Before you start

  • Android NDK installed.
  • NDK_ROOT configured to the NDK path.
  • A supported model and the documented application.

Steps

  1. Set NDK_ROOT to the installed NDK path.

  2. Run ./tools/android_build.sh.

  3. Use adb to copy the application and model file to the device, then invoke the application with adb shell as documented.

Expected outcome

An Android-targeted build that can be invoked as a command-line inference application on a device.

Guide limitations

  • The documented route is command-line use through adb; it does not describe a polished Android app.
Documentation 1 ↗
COMPARE OPTIONS

Compare options for Model inference and serving

These projects document a common task and share a project role. This is a source-based comparison, not an execution benchmark or a guarantee of interchangeability.

ProjectDocumented purposePlatformAccess and requirements
InferllmCompile a local command-line runtime for a supported quantized model.Linux, Android, CLI, Self-hostedThe public repository states that InferLLM is licensed under Apache License 2.0.
Nano Vllm V1Generate text offline from prompts using a local model.Linux, Self-hostedThe repository is public and identifies the project as MIT-licensed. It is designed for Linux and self-hosted use.
PyTorch C++ InferenceBuild a C++ executable that classifies an image using an exported TorchScript model.Linux, Docker, ServerThe repository is publicly accessible and identified as open source. No license is specified in the supplied repository details.
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
22 October 2026
Original website title
MegEngine/InferLLM · Official homepage title · retained in its source language
How we review sourcesSuggest a correction