crawl4ai-mcp

@wyattowalsh

About crawl4ai-mcp

MCP (Model Context Protocol) server wrapping Crawl4AI library as functions using Python

#python #agent #crawler #scraper #mcp #crawling #python3 #scrape #uv #uvx #agentic-workflow #agentic-ai #crawl4ai #model-context-protocol #mcp-servers #mcp-server #model-context-protocol-servers

Config

Add this server to your MCP-compatible client using the configuration below.

{
  "mcpServers": {
    "crawl4ai-mcp": {
      "command": "docker",
      "args": [
        "build",
        "-t",
        "mcp-crawl4ai",
        "."
      ]
    }
  }
}

Tools

No tools detected

We auto-extract tools from the README. The maintainer can list them under a ## Tools heading to populate this section.

Overview

What is crawl4ai-mcp?

crawl4ai-mcp is a Model Context Protocol (MCP) server that provides AI systems with access to the live web. Built on FastMCP v3 and Crawl4AI, it exposes 4 tools, 2 resources, and 3 prompts through a standardized MCP interface, backed by a lifespan-managed headless Chromium browser.

How to use crawl4ai-mcp?

Install via pip install mcp-crawl4ai or uv add mcp-crawl4ai, then run mcp-crawl4ai for stdio transport (default) or mcp-crawl4ai --transport http --port 8000 for HTTP. For Claude Desktop, add to MCP settings with the command mcp-crawl4ai and args ["--transport", "stdio"].

Key features of crawl4ai-mcp

Full MCP compliance via FastMCP v3 with tool annotations
4 focused tools for scrape, crawl, session lifecycle, and artifacts
3 prompts for common LLM workflows (summarize, extract schema, compare)
2 resources exposing server configuration and version info
Headless Chromium managed as a lifespan singleton
Multiple transports: stdio (default) and Streamable HTTP
LLM-optimized output in markdown, cleaned HTML, raw HTML, or plain text
Auto browser setup and detection of missing Playwright browsers

Use cases of crawl4ai-mcp

AI agents that need to scrape or crawl web pages for real-time information
Summarizing a page with a specified focus using the summarize_page prompt
Building a CSS extraction schema for structured data from a URL
Comparing two web pages and producing a structured comparison
Deep traversal of a website using recursive BFS/DFS from a single seed

FAQ from crawl4ai-mcp

What is the difference between crawl4ai-mcp and using Crawl4AI directly?

crawl4ai-mcp wraps Crawl4AI in an MCP server with only 2 runtime dependencies (fastmcp and crawl4ai), providing a standardized interface for AI tools and LLM clients. It adds session management, resource endpoints, and prompt templates.

What are the runtime dependencies of crawl4ai-mcp?

The server requires Python 3.13+ and has exactly two runtime dependencies: fastmcp and crawl4ai. Playwright browsers are needed for headless Chromium but are installed automatically or via mcp-crawl4ai --setup.

Does crawl4ai-mcp require a browser to be installed?

Yes, it uses headless Chromium. The server auto-detects missing Playwright browsers and attempts to install them on first startup, or you can run mcp-crawl4ai --setup manually. The Docker image includes browsers pre-installed.

What transports does crawl4ai-mcp support?

Stdio (default) for desktop clients like Claude Desktop and Cursor, and HTTP (Streamable) for remote access. HTTP binds to 127.0.0.1:8000 by default; use --host and a reverse proxy for external exposure.

Are there any known limits in crawl4ai-mcp?

The scrape tool accepts up to 20 URLs in a bounded list. The crawl tool supports both list traversal (bounded) and deep traversal (recursive BFS/DFS) from a single seed. There are configurable limits exposed via the config://server resource.

Frequently asked questions

What is the difference between crawl4ai-mcp and using Crawl4AI directly?

crawl4ai-mcp wraps Crawl4AI in an MCP server with only 2 runtime dependencies (`fastmcp` and `crawl4ai`), providing a standardized interface for AI tools and LLM clients. It adds session management, resource endpoints, and prompt templates.

What are the runtime dependencies of crawl4ai-mcp?

The server requires Python 3.13+ and has exactly two runtime dependencies: `fastmcp` and `crawl4ai`. Playwright browsers are needed for headless Chromium but are installed automatically or via `mcp-crawl4ai --setup`.

Does crawl4ai-mcp require a browser to be installed?

Yes, it uses headless Chromium. The server auto-detects missing Playwright browsers and attempts to install them on first startup, or you can run `mcp-crawl4ai --setup` manually. The Docker image includes browsers pre-installed.

What transports does crawl4ai-mcp support?

Stdio (default) for desktop clients like Claude Desktop and Cursor, and HTTP (Streamable) for remote access. HTTP binds to `127.0.0.1:8000` by default; use `--host` and a reverse proxy for external exposure.

Are there any known limits in crawl4ai-mcp?

The `scrape` tool accepts up to 20 URLs in a bounded list. The `crawl` tool supports both list traversal (bounded) and deep traversal (recursive BFS/DFS) from a single seed. There are configurable limits exposed via the `config://server` resource.

Comments

Basic information

More Other MCP servers

Codelf

unbug

A search tool helps dev to solve the naming things problem.

14.2K

DeepMark

DeepMark helps teachers deliver rapid, consistent marking with meaningful feedback for every student — in a fraction of the time. What once took a week, now takes one free period.

🚀 Model Context Protocol (MCP) Curriculum for Beginners

microsoft

This open-source curriculum introduces the fundamentals of Model Context Protocol (MCP) through real-world, cross-language examples in .NET, Java, TypeScript, JavaScript, Rust and Python. Designed for developers, it focuses on practical techniques for building modular, scalable,

16.7K

Sequential Thinking

modelcontextprotocol

Model Context Protocol Servers

88.2K

🪟 Windows-MCP

CursorTouch

MCP Server for Computer Use in Windows

6.3K