Crawlbase Web MCP Server

@crawlbase

サーバーを訪問

9 months ago

Crawlbase Web MCP Server (Model Context Protocol) connects AI agents and large language models (LLMs) with real-time web data. Built on Crawlbase’s proven web scraping and crawling infrastructure, it handles JavaScript rendering, anti-bot protection, and web data extraction at scale — powering Claude, Cursor, and Windsurf. It delivers a reliable, production-ready pipeline of live, structured data for AI workflows and intelligent applications.

概要ツールコメント

コンテンツ

Why Crawlbase MCP?

Large Language Models (LLMs) like Claude and ChatGPT are powerful, but disconnected from the live web. They can’t fetch real-time data, leading to outdated answers, hallucinations, and limited agent workflows.

Crawlbase MCP Server fixes this by bridging AI tools with fresh, structured web data — directly inside Claude Desktop, Cursor IDE, and Windsurf. Built on Crawlbase’s proven scraping infrastructure (trusted by 70,000+ developers), it delivers:

Real-time web scraping
JavaScript rendering for dynamic apps
Proxy rotation & anti-bot evasion
Structured outputs (HTML, Markdown, Screenshots)

Setup & Integration

Claude Desktop

Add to claude_desktop_config.json:

{
  "mcpServers": {
    "crawlbase": {
      "type": "stdio",
      "command": "npx",
      "args": ["@crawlbase/mcp@latest"],
      "env": {
        "CRAWLBASE_TOKEN": "your_token_here",
        "CRAWLBASE_JS_TOKEN": "your_js_token_here"
      }
    }
  }
}

Cursor IDE

Add to .cursor-settings.json:

{
  "mcpServers": {
    "crawlbase": {
      "type": "stdio",
      "command": "npx",
      "args": ["@crawlbase/mcp@latest"],
      "env": {
        "CRAWLBASE_TOKEN": "your_token_here",
        "CRAWLBASE_JS_TOKEN": "your_js_token_here"
      }
    }
  }
}

Windsurf IDE

Add to MCP settings:

{
  "mcpServers": {
    "crawlbase": {
      "type": "stdio",
      "command": "npx",
      "args": ["@crawlbase/mcp@latest"],
      "env": {
        "CRAWLBASE_TOKEN": "your_token_here",
        "CRAWLBASE_JS_TOKEN": "your_js_token_here"
      }
    }
  }
}

🔑 Get your free tokens at Crawlbase ↗.

Usage

Once configured, use these commands inside Claude, Cursor, or Windsurf:

crawl → Fetch raw HTML
crawl_markdown → Extract clean Markdown
crawl_screenshot → Capture screenshots

Example prompts:

“Crawl Hacker News and return top stories in markdown.”
“Take a screenshot of TechCrunch homepage.”
“Fetch Tesla investor relations page as HTML.”

Use Cases

Market research → Pull live data from competitors, news, and reports
E-commerce monitoring → Track products, reviews, and prices in real time
News & finance feeds → Keep AI agents up-to-date with live events
Autonomous AI agents → Give them vision to act on fresh web data

Resources & Next Steps

Looking to supercharge your AI agents with live web data? Get started here:

Playgroundで試す

サーバー設定

{
  "mcpServers": {
    "crawlbase": {
      "type": "stdio",
      "command": "npx",
      "args": [
        "@crawlbase/mcp@latest"
      ],
      "env": {
        "CRAWLBASE_TOKEN": "your_token_here",
        "CRAWLBASE_JS_TOKEN": "your_js_token_here"
      }
    }
  }
}

Build with ShipAny.