OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Developer Tools & SDKs

Synthetic Data Kit

Synthetic Data Kit is a Python command-line tool for preparing synthetic training data for large language models. It ingests files, generates QA pairs, reasoning examples or summaries with a language-model backend, curates generated examples, and exports data in formats used by fine-tuning workflows.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Synthetic Data Kit — OpenViST project profile in AI Developer Tools & SDKs.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use it when source documents need to be transformed into structured synthetic examples and exported for a separate model fine-tuning workflow.

Who it is for

  • Developers preparing synthetic examples for LLM fine-tuning
  • Teams converting document collections into structured training data

Documented uses

  • Ingest individual documents or supported directories of files.
  • Generate QA pairs, chain-of-thought examples or summaries.
  • Curate generated examples using Llama as a judge.
  • Convert curated data to a fine-tuning format such as Alpaca or OpenAI fine-tuning JSON.
Before you choose

Requirements & limits

The project is available as a public GitHub repository under the MIT license. The documented PyPI installation route specifies Python 3.10 in a Conda environment.

  • A language-model backend is needed for generating examples; vLLM and API endpoints are documented options.
  • The pipeline separates ingestion, generation, curation and export into four CLI stages.
  • The README documents individual-file and directory processing, with formats including PDF, HTML, DOCX, PPTX and TXT for directory ingestion.
  • CLI arguments and YAML configuration can be used to adjust parameters such as generation settings and curation thresholds.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Install Synthetic Data Kit

Install the CLI in a new Conda environment using the documented PyPI route.

Before you start

  • Conda
  • Python 3.10

Steps

  1. Create an environment with Python 3.10: conda create -n synthetic-data python=3.10

  2. Activate it: conda activate synthetic-data

  3. Install the package: pip install synthetic-data-kit

  4. Display the available commands with synthetic-data-kit --help

Expected outcome

The package is installed in the Conda environment and its CLI help is available.

Guide limitations

  • This covers installation and command discovery, not backend setup or dataset generation.
Documentation 1 ↗
Source-based practical guide

Prepare document-derived QA data for fine-tuning

Follow the documented single-document pipeline to ingest a PDF, create and curate QA examples, and export them.

Before you start

  • Synthetic Data Kit installed
  • A PDF document
  • A configured language-model backend for generation and curation

Steps

  1. Ingest the PDF, for example: synthetic-data-kit ingest research_paper.pdf

  2. Create QA pairs from the parsed text, for example: synthetic-data-kit create data/parsed/research_paper.txt -n 30

  3. Curate the generated examples, for example: synthetic-data-kit curate data/generated/research_paper_qa_pairs.json -t 8.5

  4. Export the curated data, for example: synthetic-data-kit save-as data/curated/research_paper_cleaned.json -f ft

Expected outcome

The pipeline produces curated examples in the documented OpenAI fine-tuning JSON format.

Guide limitations

  • The command examples rely on the documented default data paths.
  • A language-model backend is required; the documentation gives vLLM and API-endpoint configuration examples.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
11 October 2026
Original website title
meta-llama/synthetic-data-kit · Official homepage title · retained in its source language
How we review sourcesSuggest a correction