Tokenizer: tokenize and detokenize text for NLP

Tokenizer

Tokenizer is an open-source C++ and Python library, also usable through command-line clients, for tokenizing and detokenizing text. It provides Unicode-based segmentation with configurable rules, reversible output, case handling, protected sequences, and BPE and SentencePiece support.

Independent OpenViST project profile, based on linked sources.

Understand the project

Practical overview

When it fits

Use Tokenizer when an application or text-processing pipeline needs configurable token boundaries, subword processing, or a documented tokenize-and-detokenize stage.

Who it is for

Developers building text preprocessing into machine-translation or other NLP pipelines.
Python and C++ developers who need configurable tokenization and detokenization.
Users preparing text with BPE or SentencePiece subword models.

Documented uses

Split text into tokens using Unicode-based defaults or customized segmentation rules.
Preserve spacing or token-join information for later reconstruction.
Prepare text with BPE or SentencePiece subword processing.
Detokenize processed tokens back into text.

Before you choose

Requirements & limits

The repository lists an MIT license. The documented use modes are Python, C++, and command line.

Python package installation is documented as pip install pyonmttok.
Building the C++ library from source requires ICU, CMake, and a C++11-capable compiler.
The same options are exposed across Python, C++, and command-line use.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Build the C++ library and command-line clients

Compile the library and tokenization clients from source.

Before you start

ICU
CMake
A compiler supporting C++11

Steps

Initialize the Git submodule with the command below.
Documented command
```
git submodule update --init
```
Create and enter a build directory.
Run CMake with -DICU_ROOT=.
Run make to produce the dynamic library and command-line clients.

Expected outcome

The documented build produces `libOpenNMTTokenizer` and tokenization clients in `cli/`.

Guide limitations

The documented build requires ICU and the listed C++ build tools.

Documentation 1 ↗

Evidence and transparency

Sources & review

Official repository ↗

Source record details

Source availability: Active
Sources checked: 14 October 2026
Original website title: OpenNMT/Tokenizer · Official homepage title · retained in its source language

How we review sources Suggest a correction