OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
RAG & Knowledge Systems

Chonky

Chonky is a Python library that uses a fine-tuned transformer model to divide text into semantically meaningful chunks. Its ParagraphSplitter accepts text, and its MarkupRemover can strip Markdown, XML or HTML before splitting. The library is intended for tasks such as preparing documents for RAG indexing and retrieval.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Chonky — OpenViST project profile in RAG & Knowledge Systems.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use Chonky as a text-preprocessing step when a Python workflow needs semantic chunks before downstream indexing or retrieval.

Who it is for

  • Python developers building document-ingestion or RAG pipelines
  • Teams that need to segment long text by semantic boundaries before indexing or retrieval
  • Developers processing Markdown, XML or HTML who want to remove markup before chunking

Documented uses

  • Split long text into semantic chunks with ParagraphSplitter.
  • Strip supported markup formats before chunking.
  • Choose among listed transformer models, including a multilingual option.
Before you choose

Requirements & limits

  • The first run downloads the transformer model.
  • The README documents CPU use and lets users select a model identifier.
  • Listed model sequence lengths range from 512 to 1,024 tokens.
  • One of the four listed models is identified as multilingual; the other three are not.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Install Chonky and split plain text

Set up the library and pass text to its paragraph splitter.

Before you start

  • Python environment capable of installing the package
  • Network access on first use to download the transformer model

Steps

  1. Install the package with the command below.

    Documented command
    pip install chonky

  2. Import ParagraphSplitter from chonky.

  3. Create a splitter with ParagraphSplitter(device="cpu"), or specify a documented model identifier.

  4. Pass text to the splitter and iterate over the returned chunks.

Expected outcome

Text is segmented into semantic chunks that can be used as input to a RAG preprocessing workflow.

Guide limitations

  • The transformer model downloads on first use.
  • Listed models have sequence lengths of 512 or 1,024 tokens.
Documentation 1 ↗
Source-based practical guide

Remove markup before semantic splitting

Convert Markdown, XML or HTML content to text before sending it to the paragraph splitter.

Before you start

  • Installed Chonky package
  • Content in Markdown, XML or HTML

Steps

  1. Import MarkupRemover from chonky.markup_remover and ParagraphSplitter from chonky.

  2. Create a MarkupRemover and a ParagraphSplitter.

  3. Pass marked-up content to the remover.

  4. Pass the remover's text output to the splitter.

Expected outcome

Markup is removed and the resulting text is segmented into semantic chunks.

Guide limitations

  • The README identifies Markdown, XML and HTML as supported formats.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
14 October 2026
Original website title
mirth/chonky · Official homepage title · retained in its source language
How we review sourcesSuggest a correction