Crawl4ai Rag
@coleam00
About Crawl4ai Rag
No overview available yet
Config
Add this server to your MCP-compatible client using the configuration below.
{
"mcpServers": {
"crawl4ai-rag": {
"command": "python",
"args": [
"path/to/crawl4ai-mcp/src/crawl4ai_mcp.py"
],
"env": {
"TRANSPORT": "stdio",
"OPENAI_API_KEY": "your_openai_api_key",
"SUPABASE_URL": "your_supabase_url",
"SUPABASE_SERVICE_KEY": "your_supabase_service_key",
"USE_KNOWLEDGE_GRAPH": "false",
"NEO4J_URI": "bolt://localhost:7687",
"NEO4J_USER": "neo4j",
"NEO4J_PASSWORD": "your_neo4j_password"
}
}
}
}Tools
No tools detected
We auto-extract tools from the README. The maintainer can list them under a ## Tools heading to populate this section.
Overview
What is Crawl4ai Rag?
A Model Context Protocol (MCP) server that integrates Crawl4AI web crawling with Supabase vector storage to provide AI agents and coding assistants with web crawling and RAG (Retrieval-Augmented Generation) capabilities.
How to use Crawl4ai Rag?
Install via Docker or uv (Python 3.12+), configure a .env file with Supabase and OpenAI API key, set up a Supabase database with pgvector, and optionally set up Neo4j for knowledge graph features. The server exposes MCP tools for crawling, searching, and querying crawled content.
Key features of Crawl4ai Rag
- Smart URL detection (regular pages, sitemaps, text files)
- Recursive crawling following internal links
- Parallel processing for efficient multi-page crawling
- Content chunking by headers and size
- Vector search with optional source filtering
- Advanced RAG strategies (contextual embeddings, hybrid search, agentic RAG, reranking, knowledge graph)
Use cases of Crawl4ai Rag
- AI coding assistants scraping and indexing documentation for code-aware RAG
- Building a knowledge engine for AI agents to crawl and query websites
- Performing semantic search over previously crawled content with source filtering
- Detecting AI hallucinations by validating code against a repository knowledge graph
FAQ from Crawl4ai Rag
What dependencies does Crawl4ai Rag require?
Requires Docker or Python 3.12+, a Supabase project (for RAG), and an OpenAI API key. Neo4j is optional for knowledge graph functionality.
How are RAG strategies configured?
Each strategy (contextual embeddings, hybrid search, agentic RAG, reranking, knowledge graph) is enabled via a USE_* environment variable set to "true" in the .env file. All default to "false".
Where is crawled content stored?
Content is stored in a Supabase database with the pgvector extension. A separate table is created for agentic RAG code examples if enabled. Knowledge graph data is stored in a Neo4j graph database.
What transport does the server use?
The server uses SSE (Server-Sent Events) transport by default, configurable via the TRANSPORT environment variable.
Are there known limitations or planned improvements?
The repository is a testbed for integration into Archon; known areas for improvement include supporting multiple embedding models, local Ollama execution, faster crawling/indexing, and enhanced chunking strategies.
Frequently asked questions
What dependencies does Crawl4ai Rag require?
Requires Docker or Python 3.12+, a Supabase project (for RAG), and an OpenAI API key. Neo4j is optional for knowledge graph functionality.
How are RAG strategies configured?
Each strategy (contextual embeddings, hybrid search, agentic RAG, reranking, knowledge graph) is enabled via a `USE_*` environment variable set to `"true"` in the `.env` file. All default to `"false"`.
Where is crawled content stored?
Content is stored in a Supabase database with the pgvector extension. A separate table is created for agentic RAG code examples if enabled. Knowledge graph data is stored in a Neo4j graph database.
What transport does the server use?
The server uses SSE (Server-Sent Events) transport by default, configurable via the `TRANSPORT` environment variable.
Are there known limitations or planned improvements?
The repository is a testbed for integration into Archon; known areas for improvement include supporting multiple embedding models, local Ollama execution, faster crawling/indexing, and enhanced chunking strategies.
Basic information
More Memory & Knowledge MCP servers
Rust Docs MCP Server
Govcraft🦀 Prevents outdated Rust code suggestions from AI assistants. This MCP server fetches current crate docs, uses embeddings/LLMs, and provides accurate context via a tool call.
MemoryMesh
CheMiguel23A knowledge graph server that uses the Model Context Protocol (MCP) to provide structured memory persistence for AI models.
Zettelkasten MCP Server
entanglrA Model Context Protocol (MCP) server that implements the Zettelkasten knowledge management methodology, allowing you to create, link, explore and synthesize atomic notes through Claude and other MCP-compatible clients.
Dash Api Docs Mcp Server
KapeliMCP server for Dash, the macOS API documentation browser
Mcp Knowledge Graph
shanehollomanMCP server enabling persistent memory for Claude through a local knowledge graph - fork focused on local development
Comments