NOTICE

@BjornMelin

About NOTICE

🚀 High-performance MCP Server for Crawl4AI - Enable AI assistants to access web scraping, crawling, and deep research via Model Context Protocol. Faster and more efficient than FireCrawl!

Config

No standard config provided

This server doesn't expose a parseable MCP config block in its README. See the repository for install instructions.

Repository

Tools

Crawl web pages from a starting URL

Retrieve crawl data by ID

List all crawls or filter by domain

Search indexed documents by query

Extract structured content from a URL

Overview

What is Crawl4AI MCP Server?

Crawl4AI MCP Server is an open-source implementation of the Model Context Protocol (MCP) that integrates with the Crawl4AI web scraping and crawling library. It is deployed as a remote MCP server on CloudFlare Workers, enabling AI assistants such as Claude to perform web scraping, crawling, and structured data extraction.

How to use Crawl4AI MCP Server?

Install Node.js (v18+), npm, and Wrangler, then clone the repository, run npm install, create a CloudFlare KV namespace, update wrangler.toml, and deploy with npm run deploy. Connect MCP clients (e.g., Claude Desktop) using the assigned CloudFlare Workers URL. Available tools include crawl, getCrawl, listCrawls, search, and extract.

Key features of Crawl4AI MCP Server

Single webpage scraping and full website crawling
Configurable crawl depth and page limits
Asynchronous crawling for efficient site traversal
Deep research across multiple pages
Structured data extraction using CSS selectors or LLM extraction
OAuth and API key (Bearer token) authentication

Use cases of Crawl4AI MCP Server

AI assistants retrieving live web content on demand
Automated deep research of multiple linked pages
Extracting specified data fields from web pages for analysis
Building a searchable index of crawled content

FAQ from Crawl4AI MCP Server

Is this server ready for production use?

No. The README explicitly states the server is under development and not ready for production use.

What are the prerequisites for running the server?

You need Node.js v18+, npm, the Wrangler CLI, and a CloudFlare account. A CloudFlare KV namespace is required for storing crawl data.

How is crawled data stored?

Data is stored in a CloudFlare KV namespace configured in wrangler.toml under the binding CRAWL_DATA.

What authentication methods are supported?

The server supports both OAuth authentication (via workers-oauth-provider) and API key authentication using Bearer tokens.

Which MCP clients are compatible?

The server implements the standard Model Context Protocol and can be used with any MCP client, such as Claude Desktop.

Frequently asked questions

Is this server ready for production use?

No. The README explicitly states the server is under development and not ready for production use.

What are the prerequisites for running the server?

You need Node.js v18+, npm, the Wrangler CLI, and a CloudFlare account. A CloudFlare KV namespace is required for storing crawl data.

How is crawled data stored?

Data is stored in a CloudFlare KV namespace configured in `wrangler.toml` under the binding `CRAWL_DATA`.

What authentication methods are supported?

The server supports both OAuth authentication (via workers-oauth-provider) and API key authentication using Bearer tokens.

Which MCP clients are compatible?

The server implements the standard Model Context Protocol and can be used with any MCP client, such as Claude Desktop.

Comments

Basic information

More Browser Automation MCP servers

MCP Browser Kit

ndthanhdev

An MCP Server that enables AI assistants to interact with your local browsers.

Apify Model Context Protocol (MCP) Server

apify

The Apify MCP server enables your AI agents to extract data from social media, search engines, maps, e-commerce sites, or any other website using thousands of ready-made scrapers, crawlers, and automation tools available on the Apify Store.

1.7K