OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
RAG & Knowledge Systems

Llama 2 Open Source LLM Cpu Inference

A MIT-licensed Python command-line application and guide for answering questions about ingested documents with quantized language models running locally on a CPU. It combines document embeddings and FAISS retrieval with a Llama 2 chat model through C Transformers and LangChain.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Llama 2 Open Source LLM Cpu Inference — OpenViST project profile in RAG & Knowledge Systems.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use it as a starting point when you want a Python command-line workflow for document Q&A with a quantized model on CPU rather than a hosted inference service.

Who it is for

  • Developers building local document question-answering workflows.
  • Teams exploring self-managed CPU inference for queries over internal or organizational documents.

Documented uses

  • Ask natural-language questions about an ingested document collection.
  • Create a FAISS vector store from a dataset using the documented ingestion script.
  • Adapt the example annual-report workflow to an organization’s own documents.
Before you choose

Requirements & limits

The public GitHub repository specifies an MIT license. The documented application is self-hosted and command-line based.

  • The example requires a GGML model binary in the models/ directory.
  • The documented stack includes Python, C Transformers, GGML, LangChain, FAISS, Sentence-Transformers and Poetry.
  • The repository provides separate scripts for creating the vector store and submitting a query.
  • Using a different quantized model is described broadly, but individual alternatives are not documented or tested.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Run a command-line question against the included document workflow

Pass a document question to the application using the documented command-line entry point.

Before you start

  • The project files and Python dependencies.
  • A GGML model binary downloaded from the linked Llama 2 model location and placed in the models/ folder.
  • Poetry for the README’s example command; the README says to omit poetry run if not using Poetry.

Steps

  1. Place the downloaded GGML model binary in models/.

  2. Open a terminal in the project directory.

  3. Run poetry run python main.py "", or omit poetry run if not using Poetry.

Expected outcome

The application accepts a user question through `main.py` for document Q&A.

Guide limitations

  • The README documents a sample dataset and query, but does not establish support for other document formats.
  • The README does not specify hardware performance or response-time expectations.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
16 October 2026
Original website title
kennethleungty/Llama-2-Open-Source-LLM-CPU-Inference · Official homepage title · retained in its source language
How we review sourcesSuggest a correction