OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Infrastructure & Serving

Jlama

Jlama is a Java-based engine for running large language model inference, either embedded in a Java application or through a command-line client with chat, completion and REST API commands.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Jlama — OpenViST project profile in AI Infrastructure & Serving.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use Jlama when you want model inference as part of a Java application or need its CLI for local model interaction. The repository also documents commands for distributed inference.

Who it is for

  • Java developers adding model inference to an application
  • Developers who want a command-line interface for local model chat or prompt completion
  • Teams exploring distributed inference through coordinator and worker commands

Documented uses

  • Generate model responses from Java application code
  • Chat with a model or complete prompts using the CLI
  • Expose a local OpenAI-compatible REST API for a model
  • Download, list, quantize and remove local models
  • Run distributed inference using coordinator and worker commands
Before you choose

Requirements & limits

The public GitHub repository identifies the project as open source under the Apache-2.0 license.

  • The README lists Java 20 or later as a requirement; its Maven setup instructions refer to Java 21 preview features.
  • Panama Vector is the default backend; Native SIMD is optional and WebGPU is marked experimental.
  • The listed model families include Gemma, Llama, Mistral, Mixtral, Qwen2, IBM Granite, GPT-2 and BERT.
  • The project documents both embedded Java use and a CLI with local REST API and distributed-inference commands.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Install the Jlama CLI and start a local model chat

Use JBang to install Jlama, download the named model automatically and start its local chat interface.

Before you start

  • JBang installed
  • A model identifier; the example uses tjake/Llama-3.2-1B-Instruct-JQ4

Steps

  1. Install or set up JBang using its installation instructions.

  2. Install the CLI with jbang app install --force jlama@tjake and respond to its source-trust prompt.

  3. Run jlama restapi tjake/Llama-3.2-1B-Instruct-JQ4 --auto-download.

  4. Open http://localhost:8080/ in a browser.

Expected outcome

The CLI starts a local chat API and UI for the specified model.

Guide limitations

  • The documented example is for local use and does not specify deployment or network exposure settings.
  • The installation prompt asks whether to trust the source.
Documentation 1 ↗
Source-based practical guide

Add Jlama to a Java project

Set up the documented Maven dependencies and Java runtime options for using Jlama in an application.

Before you start

  • A Java project using Maven
  • Java 20 or later; the Java-project instructions specify Java 21 preview features
  • A model identifier and working directory for the model

Steps

  1. Add the com.github.tjake:jlama-core and com.github.tjake:jlama-native dependencies shown in the README, including the native artifact classifier.

  2. Enable preview features and the incubator vector module with JDK_JAVA_OPTIONS="--add-modules jdk.incubator.vector --enable-preview", or configure the Maven compiler and failsafe plugins.

  3. Follow the README’s Java example to download the Hugging Face model, load it, build a prompt context and call the generator.

Expected outcome

The documented setup provides the dependencies and runtime configuration for embedding Jlama inference in a Java application.

Guide limitations

  • The README notes Java 20 or later in its requirements section but the project-specific instructions refer to Java 21 preview features.
  • Model loading and generation details depend on the chosen model and application code.
Documentation 1 ↗
COMPARE OPTIONS

Compare options for Model inference and serving

These projects document a common task and share a project role. This is a source-based comparison, not an execution benchmark or a guarantee of interchangeability.

ProjectDocumented purposePlatformAccess and requirements
JlamaGenerate model responses from Java application codeServerThe public GitHub repository identifies the project as open source under the Apache-2.0 license.
MorphingdbImport a libtorch model and create predictions from values in a PostgreSQL table.Docker, Linux, Self-hosted, ServerThe README documents a Docker image and a build-from-repository option, followed by connecting to PostgreSQL and enabling the pgdl extension.
Gateway — pyadavSend chat-completion requests to a provider through a common server endpoint.ServerThe repository is public and identifies the project as Apache-2.0 licensed. README installation instructions include binaries, platform package managers and Docker.
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
15 October 2026
Original website title
tjake/Jlama · Official homepage title · retained in its source language
How we review sourcesSuggest a correction