Practical overview
When it fits
Use DegreeGuru as a codebase and documented RAG workflow when you want to turn web content into a searchable knowledge base for a chatbot and can configure its crawler, vector service and model credentials.
Who it is for
- Developers building a question-answering chatbot grounded in a curated collection of web pages.
- Teams exploring retrieval-augmented generation for academic or other domain-specific material.
Documented uses
- Crawl configured web pages and split their text into chunks.
- Create embeddings and store them in Upstash Vector with source text and URL metadata.
- Build a chatbot that retrieves indexed material to support answers.
Requirements & limits
The project is available as a public GitHub repository. The documented setup requires Upstash Vector credentials and an OpenAI API key.
- The repository documents both a Scrapy content-ingestion component and a Next.js chatbot, so the crawler and query interface are distinct parts of the project.
- The README specifies OpenAI gpt-3.5-turbo-1106 for generation and text-embedding-ada-002 for embeddings, with a 1536-dimensional Upstash Vector database.
- The project is described as community-maintained; Upstash may contribute but does not officially support or assume responsibility for it.
- Changing the corpus is described as domain-agnostic in principle, but the repository does not show ingestion steps or test results for an alternative corpus.
Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.
Configure a web-content crawler for a knowledge base
Prepare DegreeGuru's crawler to collect and index pages from a chosen website.
Before you start
- A fork or local clone of the DegreeGuru repository.
- An Upstash Vector database with 1536 dimensions.
- Upstash Vector REST URL and token, plus an OpenAI API key.
- Python environment and the crawler's required libraries.
Steps
Set the Upstash Vector and OpenAI credentials as environment variables.
In the degreegurucrawler directory, create and activate a Python virtual environment, then install the libraries from requirements.txt.
Edit degreegurucrawler/utils/crawler.yaml to set start URLs, link allow/deny rules, embedding model and text-splitting parameters.
Review the crawler's settings.py configuration before proceeding to the README's crawler-running instructions.
Expected outcome
The crawler configuration and dependencies are prepared for the documented crawling stage, which extracts page text, creates embeddings and stores vectors with source text and URL metadata.
Guide limitations
- This guide covers configuration only; it does not claim that the crawler was run.
- The README's subsequent crawler execution instructions are not included here.
Compare options for Vector search
These projects document a common task and share a project role. This is a source-based comparison, not an execution benchmark or a guarantee of interchangeability.
| Project | Documented purpose | Platform | Access and requirements |
|---|---|---|---|
| DegreeGuru | Crawl configured web pages and split their text into chunks. | Cloud, Web | The project is available as a public GitHub repository. The documented setup requires Upstash Vector credentials and an OpenAI API key. |
| Chatbot with PDF for Semantic Search | Prepare document content as embeddings and save it to a supported vector database. | Web | The repository is identified as open source and publicly visible on GitHub. |
| AI LLM Playground | Ask questions about PDF documents through an assistant interface. | Web |
Sources & review
Source record details
- Source availability
- Active
- Sources checked
- 16 October 2026
- Original website title
- upstash/degree-guru · Official homepage title · retained in its source language