OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Data & Analytics

Petastorm

Petastorm is a library for accessing machine-learning datasets stored in Apache Parquet for single-machine or distributed deep-learning training and evaluation.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Petastorm — OpenViST project profile in AI Data & Analytics.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use Petastorm when a deep-learning workflow needs to read datasets stored in Apache Parquet, whether the work runs on one machine or uses distributed processing.

Who it is for

  • Developers building TensorFlow or PyTorch training and evaluation workflows around Parquet datasets.
  • Teams using PySpark for distributed deep-learning data pipelines.
  • Python practitioners who need to access Parquet data for deep-learning work.

Documented uses

  • Read Parquet-backed datasets as input to TensorFlow training or evaluation.
  • Load Parquet data into PyTorch training and evaluation workflows.
  • Connect Parquet datasets to distributed model-training pipelines using PySpark.
  • Access Parquet data from Python code for deep-learning evaluation.
Before you choose

Requirements & limits

  • The project is a data-access library, not a model-training application or dataset host.
  • The described workflow supports Apache Parquet data and names TensorFlow, PyTorch, PySpark and pure Python integrations.
  • The stated use spans both single-machine and distributed training and evaluation.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Install Petastorm with optional dependencies

Install Petastorm, optionally including the documented OpenCV and GPU TensorFlow extras.

Before you start

  • Python environment with pip available.

Steps

  1. Install the base package with the command below.

    Documented command
    pip install petastorm

  2. To include OpenCV and the GPU TensorFlow extra, use the command below instead.

    Documented command
    pip install petastorm[opencv,tf_gpu]

Expected outcome

Petastorm is installed; the optional command requests the OpenCV and GPU TensorFlow extras.

Guide limitations

  • Other extras listed by the project are tf, torch, opencv, docs, and test; extras are not installed automatically.
  • This installation guidance does not document how to create or load a dataset.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗Guide documentation ↗
Source record details
Source availability
Active
Sources checked
11 October 2026
Original website title
GitHub - uber/petastorm: Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and · Official homepage title · retained in its source language
How we review sourcesSuggest a correction