Practical overview
When it fits
Use Warc Gpt when you have WARC files and want a self-managed retrieval and question-answering workflow for their extracted HTML and PDF text.
Who it is for
- Researchers exploring material preserved in web archives
- Archive teams building an AI-assisted interface for WARC collections
- Developers adapting a Python retrieval-augmented generation application
Documented uses
- Ingest WARC files and create a searchable knowledge base from HTML and PDF response text.
- Search archive excerpts and inspect associated WARC record details.
- Ask questions about an ingested collection through a web interface.
- Use the REST API to search records or generate text completions.
- Visualize stored embeddings in a basic interactive two-dimensional plot.
Requirements & limits
- Requires Python 3.11 or later and Poetry, with a documented venv-and-pip alternative.
- Requires at least one inference option: OpenAI API or Ollama; compatible providers can be configured through OpenAI-compatible settings.
- The documented ingestion extracts text from HTML and PDF response records and clears
./chromadbwhen run. - The repository identifies the project as experimental and lists an MIT license.
Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.
Install and configure Warc Gpt
Prepare the application and configure a supported inference provider.
Before you start
- Python 3.11 or later
- Python Poetry, or Python's venv and pip as the documented alternative
- Access to an OpenAI API or an Ollama service; at least one is required
Steps
Clone the repository and enter its directory.
Install dependencies with Poetry, or create a virtual environment and install the project with pip.
Copy
.env.exampleto.envand edit the settings.Configure OpenAI API or Ollama; for an OpenAI-compatible provider, set
OPENAI_BASE_URLandOPENAI_COMPATIBLE_MODEL.
Expected outcome
The application dependencies are installed and its environment-based settings are configured.
Guide limitations
- This guide covers installation and configuration only, not a verified deployment.
- At least one inference option is needed; the application defaults to trying Ollama at
http://localhost:11434.
Ingest WARC files for archive search
Populate the application's knowledge base from WARC files for retrieval and question answering.
Before you start
- Warc Gpt installed and configured with at least one inference option
- WARC files placed in the
./warcdirectory
Steps
Place the WARC files to explore under
./warc.Run
poetry run flask ingestfrom the project directory.
Expected outcome
The application extracts text from HTML and PDF response records, creates embeddings and stores them in its vector store for use as a knowledge base.
Guide limitations
- Ingestion clears the
./chromadbfolder. - The documented extraction covers
text/htmlandapplication/pdfresponse records.
Compare options for Retrieval-augmented applications
These projects document a common task and share a project role. This is a source-based comparison, not an execution benchmark or a guarantee of interchangeability.
| Project | Documented purpose | Platform | Access and requirements |
|---|---|---|---|
| Warc Gpt | Ingest WARC files and create a searchable knowledge base from HTML and PDF response text. | Server, Web | |
| Rag Powered AI Assistant Transforming Data Retrieval And Analysis Across The Web And Pdfs | Retrieve relevant passages from web and PDF content to answer a question. | Web | |
| LangChain in Supabase Edge Functions | Initialize and start a Supabase project locally. | Cloud, Server |
Sources & review
Source record details
- Source availability
- Active
- Sources checked
- 15 October 2026
- Original website title
- harvard-lil/warc-gpt · Official homepage title · retained in its source language