OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Developer Tools & SDKs

Sentencepiece

SentencePiece is an Apache-2.0 C++ and Python library that trains and applies text-tokenization models for neural text generation. It supports BPE and unigram subword units, can train directly from raw sentences, and encodes text as subword pieces or vocabulary IDs for later decoding.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Sentencepiece — OpenViST project profile in AI Developer Tools & SDKs.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use SentencePiece when a neural text-generation or translation pipeline needs a trainable subword tokenizer that works from raw Unicode sentences without language-specific pre-tokenization.

Who it is for

  • Developers preparing text for neural text-generation systems with a fixed vocabulary.
  • NLP and machine-translation teams building tokenization into training or inference pipelines.
  • Practitioners processing languages where whitespace is not a reliable word boundary.

Documented uses

  • Train a subword tokenizer from raw text.
  • Convert text into token pieces or vocabulary IDs for a model pipeline.
  • Decode token pieces or IDs back into text.
  • Use a consistent tokenizer across C++ and Python workflows with the same model file.
Before you choose

Requirements & limits

  • The documented Python quick start requires a raw text file and a chosen vocabulary size.
  • SentencePiece supports both BPE and unigram subword models.
  • Whitespace is represented in the tokenization, supporting text reconstruction.
  • The same model file contains normalization rules, vocabulary mapping and segmentation data for consistent results across C++ and Python.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Train a Python tokenizer and encode text

Create a SentencePiece model from a raw text file, then encode and decode a sample string.

Before you start

  • Python and pip
  • A raw text training file

Steps

  1. Install the Python module with the command below.

    Documented command
    pip install sentencepiece

  2. Train a model from the text file with SentencePieceTrainer.train, setting the input file, model prefix and vocabulary size.

  3. Load the resulting .model file with SentencePieceProcessor.

  4. Encode text as subword strings or integer IDs, then decode those values.

Expected outcome

A trained tokenizer model and an example of encoding text and reconstructing it from its pieces or IDs.

Guide limitations

  • The example requires a training text file and sets a vocabulary size.
  • This is a library workflow for use in Python code, not a hosted service.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
13 October 2026
Original website title
google/sentencepiece · Official homepage title · retained in its source language
How we review sourcesSuggest a correction