OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
RAG & Knowledge Systems

Gcrawlai

GcrawlAI is an open-source web crawling and extraction application that converts website pages into clean Markdown, HTML, structured metadata, links, and screenshots for use in RAG pipelines, LLM applications, and semantic-search indexes.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Gcrawlai — OpenViST project profile in RAG & Knowledge Systems.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use GcrawlAI when a project needs to collect and structure website content before feeding it into a RAG, LLM, or semantic-search workflow.

Who it is for

  • Developers preparing website content for retrieval-augmented generation systems.
  • Teams building LLM applications or semantic-search indexes from web content.
  • Python developers who want to request page scraping or multi-page crawling through an SDK.

Documented uses

  • Extract a single web page into Markdown or other supported formats.
  • Crawl a site with a configured page cap and domain scope.
  • Collect page metadata, discovered links, and full-page screenshots.
  • Prepare cleaned website content for downstream AI and search systems.
Before you choose

Requirements & limits

The project is open source under the MIT license. The README lists some custom extractors as paid-plan features.

  • The documented SDK examples require an API key.
  • Self-hosting prerequisites listed in the README include Python 3.9+, PostgreSQL, Redis, and Git.
  • The documented architecture uses FastAPI, Playwright, Celery, Redis, PostgreSQL, and Angular.
  • Certain custom retail and travel extractors are described as paid-plan features and reside in a closed-source enterprise repository.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Extract one page as Markdown with the Python SDK

Use the documented SDK example to request Markdown from a single website URL.

Before you start

  • Python environment with the gcrawl-sdk package installed using pip.
  • A Gcrawl API key.
  • A target page URL.

Steps

  1. Install the SDK with the command below.

    Documented command
    pip install gcrawl-sdk

  2. Import GcrawlClient from gcrawl_sdk and create a client with your API key.

  3. Call client.scrape with the target URL, formats=["markdown"], and wait=True.

  4. Read the result from result.markdown.

Expected outcome

The example returns the page's Markdown through the SDK result.

Guide limitations

  • The example demonstrates a single-page request, not a multi-page crawl.
  • The source does not specify account or pricing requirements.
Documentation 1 ↗
Source-based practical guide

Configure a bounded, same-domain crawl

Set the documented crawl options for a multi-page crawl with a page cap and same-domain restriction.

Before you start

  • A starting homepage or domain URL.
  • An API request to the documented POST /api/v1/crawl endpoint.

Steps

  1. Set the required starting url.

  2. Set crawl.max_pages to the desired page cap.

  3. Keep crawl.same_domain_only enabled to restrict the crawl to the base domain.

  4. Leave crawl.include_subdomains disabled unless subdomains should be included.

Expected outcome

The request configuration specifies a background crawl with a page limit and domain scope.

Guide limitations

  • The README describes crawl initiation as asynchronous and does not provide a complete request payload or execution result in the cited configuration section.
Documentation 1 ↗
COMPARE OPTIONS

Compare options for Retrieval-augmented applications

These projects document a common task and share a project role. This is a source-based comparison, not an execution benchmark or a guarantee of interchangeability.

ProjectDocumented purposePlatformAccess and requirements
GcrawlaiExtract a single web page into Markdown or other supported formats.Cloud, Self-hosted, ServerThe project is open source under the MIT license. The README lists some custom extractors as paid-plan features.
Rag Powered AI Assistant Transforming Data Retrieval And Analysis Across The Web And PdfsRetrieve relevant passages from web and PDF content to answer a question.Web
LangChain in Supabase Edge FunctionsInitialize and start a Supabase project locally.Cloud, Server
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
15 October 2026
Original website title
GramosoftAI/GcrawlAI · Official homepage title · retained in its source language
How we review sourcesSuggest a correction