OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Data & Analytics

Data Juicer

Data Juicer is an open-source Python data-processing toolkit that uses modular operators and YAML recipes to clean, synthesize, and analyze foundation-model datasets. It supports text, image, audio, video, and multimodal workflows, with use cases including pretraining, instruction tuning, agent interaction traces, and RAG data.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Data Juicer — OpenViST project profile in AI Data & Analytics.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use Data Juicer when a foundation-model workflow needs configurable dataset preparation or analysis, particularly when the data includes multiple modalities or the steps need to be composed into a reusable pipeline.

Who it is for

  • Developers assembling repeatable data-processing pipelines for foundation-model projects.
  • Researchers and data teams preparing or analyzing multimodal datasets.
  • Teams curating instruction-tuning examples, agent traces, or RAG data.

Documented uses

  • Filter and normalize text samples through composable operators.
  • Prepare and analyze data for pretraining and instruction tuning.
  • Clean or structure agent interaction traces.
  • Prepare and profile datasets for RAG workflows.
  • Build reusable processing workflows with YAML recipes or Python.
Before you choose

Requirements & limits

The project is available as a public GitHub repository under the Apache-2.0 license. The documented package installation uses `uv`.

  • Choose between the documented command-line configuration workflow and Python API according to whether the job is recipe-based or code-driven.
  • Check the operator documentation for support relevant to the dataset and transformations you need.
  • The README describes both local-scale examples and large-scale processing claims; the small Python example is not a deployment recipe.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Install Data Juicer and run the documented sample configuration

Install the Python package and start a processing job using the sample YAML configuration.

Before you start

  • Python package installation using uv
  • The Data Juicer repository or its sample configuration

Steps

  1. Install the package with the command below.

    Documented command
    uv pip install py-data-juicer

  2. Run dj-process --config demos/process_simple/process.yaml.

Expected outcome

The documented sample processing configuration is submitted to the Data Juicer command-line processor.

Guide limitations

  • The README does not specify the sample job's output or runtime requirements.
Documentation 1 ↗
Source-based practical guide

Apply filter and normalization operators to a Python dataset

Create a small text dataset and process it with the documented length filter and whitespace-normalization mapper.

Before you start

  • Python environment with Data Juicer installed
  • The NestedDataset, TextLengthFilter, and WhitespaceNormalizationMapper components

Steps

  1. Create a NestedDataset from a dictionary containing text samples.

  2. Call process with TextLengthFilter(min_len=10) followed by WhitespaceNormalizationMapper().

  3. Iterate over the returned dataset to print its samples.

Expected outcome

The example applies a minimum-text-length filter and whitespace normalization to the dataset.

Guide limitations

  • The example covers a small text dataset and does not document a multimodal or distributed run.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
7 October 2026
Original website title
datajuicer/data-juicer · Official homepage title · retained in its source language
How we review sourcesSuggest a correction