OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
RAG & Knowledge Systems

Discovai Crawl

DiscovAI Crawl is an open-source TypeScript web-crawling API that retrieves URL content and can return cleaned HTML, Markdown and plain text, along with LLM-generated summaries, extracted details, embeddings and website screenshots.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Discovai Crawl — OpenViST project profile in RAG & Knowledge Systems.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use it when a development workflow needs to turn URLs into processed text and structured page data, or prepare embeddings and screenshots for downstream systems.

Who it is for

  • Developers building URL ingestion for AI tools or knowledge systems
  • Teams preparing crawled page content for vector-database workflows
  • Developers who need structured page summaries, feature lists, FAQs or screenshots

Documented uses

  • Convert webpage content into cleaned HTML, Markdown or plain text.
  • Extract summaries, features and FAQs from crawled pages.
  • Use returned embeddings as inputs to a vector database.
  • Save webpage screenshots to AWS S3 or Cloudflare R2.
Before you choose

Requirements & limits

  • The README describes the project as a work in progress.
  • Local setup uses pnpm and includes a Playwright installation step.
  • AWS S3 and Cloudflare R2 are named for screenshot storage.
  • The README says embeddings are ready for vector-database integration but does not identify a specific database.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Set up the local development environment

Install the project dependencies and Playwright, then start the documented local development server.

Before you start

  • The project repository and pnpm
  • A local environment capable of running the documented commands

Steps

  1. Run pnpm i from the project directory.

  2. Run cd apps/api && pnpm exec playwright install.

  3. Run pnpm dev.

  4. Open http://localhost:3000.

Expected outcome

The README's local development server is started and the local address is opened.

Guide limitations

  • The README marks the project as a work in progress.
  • The documentation section is TODO; these steps do not document production deployment or URL-submission details.
Documentation 1 ↗
COMPARE OPTIONS

Compare options for Vector search

These projects document a common task and share a project role. This is a source-based comparison, not an execution benchmark or a guarantee of interchangeability.

ProjectDocumented purposePlatformAccess and requirements
Discovai CrawlConvert webpage content into cleaned HTML, Markdown or plain text.Self-hosted, Server
MettisIndex text documents through an HTTP endpoint.Linux, Self-hosted, Server
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
17 October 2026
Original website title
DiscovAI/DiscovAI-crawl · Official homepage title · retained in its source language
How we review sourcesSuggest a correction