Practical overview
When it fits
Use Trankit when a Python workflow needs sentence- or document-level linguistic analysis across supported languages, or when a team wants to train a customized tokenization and sentence-splitting pipeline.
Who it is for
- Python developers adding language analysis to text-processing applications
- Researchers and practitioners working with multilingual NLP
- Teams that need to train customized tokenization or sentence-splitting pipelines
Documented uses
- Add sentence segmentation and tokenization to text workflows.
- Generate part-of-speech, morphological, dependency, or named-entity annotations.
- Process text across languages using language-specific pretrained pipelines or automatic language detection.
- Train a customized tokenization and sentence-splitting pipeline using prepared training and development data.
Requirements & limits
The project is open source under the Apache-2.0 license. Its README currently advises installation from source because of a server problem and says model files are downloaded from its Hugging Face repository.
- The README describes trainable pipelines for more than 100 languages and 90 pretrained pipelines for 56 languages.
- Auto Mode detects input language and selects corresponding language-specific models; manual language selection remains available.
- Input can be raw or pretokenized and processed at sentence or document level, but pretokenized input may be unsuitable for languages requiring multi-word token expansion.
- The documented custom-training example covers token and sentence splitting and uses text and CoNLL-U training and development files.
- The README reports comparisons with Stanza, spaCy, and UDPipe in its performance documentation; those results are tied to the tasks and datasets described there.
Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.
Install Trankit from source
Set up the toolkit from its repository, following the project's current installation guidance.
Before you start
- Git
- Python and pip
- Internet access to retrieve the repository and model files
Steps
Clone the Trankit repository: git clone https://github.com/nlp-uoregon/trankit.git
Change into the trankit directory: cd trankit
Install the package in editable mode: pip install -e .
Expected outcome
Trankit is installed from the cloned source; the README says this route downloads models from the project's Hugging Face models repository.
Guide limitations
- The README reports a server problem and advises source installation for now rather than pip installation.
- This setup recipe does not cover initializing or running a pipeline.
Process raw or pretokenized text with a pretrained pipeline
Initialize an English pretrained pipeline and submit sentence or document text for processing.
Before you start
- Trankit installed from source
- Python
- Internet access for the initial pretrained-model download, unless the models are already cached
Steps
Import Pipeline from trankit and initialize it with Pipeline(lang='english', gpu=True, cache_dir='./cache').
Pass raw text or pretokenized input to the pipeline for document processing.
For sentence-level processing, pass the sentence with is_sent=True.
Expected outcome
The pipeline returns processed output for the submitted sentence or document.
Guide limitations
- The example configures GPU use; the supplied instructions do not establish hardware compatibility or performance.
- For languages requiring multi-word token expansion, pretokenized input may not be appropriate; consult the language table.
- The README does not provide a complete output schema in the available instructions.
Sources & review
Source record details
- Source availability
- Active
- Sources checked
- 13 October 2026
- Original website title
- nlp-uoregon/trankit · Official homepage title · retained in its source language