olostep-mcp

Q: How do I get an Olostep API key?

Sign up at [olostep.com/auth](https://olostep.com/auth) to obtain your API key.

@olostep

About olostep-mcp

MCP server for Olostep — the web scraping, crawling, and search infrastructure used by top AI companies. Gives any MCP-compatible AI agent the ability to scrape, crawl, batch-extract, and search the web in real time.

#web-scraping #content-extraction #ai-assistants

Config

Add this server to your MCP-compatible client using the configuration below.

{
  "mcpServers": {
    "olostep-mcp": {
      "command": "npx",
      "args": [
        "-y",
        "olostep-mcp"
      ],
      "env": {
        "OLOSTEP_API_KEY": "YOUR_API_KEY_HERE"
      }
    }
  }
}

Tools

No tools detected

We auto-extract tools from the README. The maintainer can list them under a ## Tools heading to populate this section.

Overview

What is olostep-mcp?

A Model Context Protocol (MCP) server that integrates with Olostep for web scraping, content extraction, and web search. It requires an Olostep API key (obtainable at olostep.com/auth). Designed for developers and AI agents needing structured data from websites.

How to use olostep-mcp?

Install via npx (npx -y olostep-mcp), Docker (docker pull olostep/mcp-server), or connect to the remote endpoint (https://mcp.olostep.com/mcp). Set the OLOSTEP_API_KEY environment variable. Configure in your MCP client (Cursor, Claude Desktop, Windsurf, VS Code, Metorial) using the provided JSON examples.

Key features of olostep-mcp

Scrape websites in HTML, Markdown, JSON, or Plain Text
Parser-based web search with structured results
AI Answers with citations and optional JSON-shaped outputs
Batch scraping of up to 10,000 URLs
Autonomous site crawling from a start URL
Website URL discovery with include/exclude filters
Country-specific request routing for geo-targeted content
Configurable wait times for JavaScript-heavy websites

Use cases of olostep-mcp

Extract clean markdown from a single documentation page
Search the web and return structured data (e.g., product listings)
Get AI-powered answers with cited sources for research
Batch scrape thousands of URLs for large-scale data collection
Crawl an entire website to map its structure and content

FAQ from olostep-mcp

How do I get an Olostep API key?

What output formats are supported?

Scraped content can be returned as HTML, Markdown, JSON, or Plain Text (configurable per request).

Can I scrape multiple URLs at once?

Yes, use the batch_scrape_urls tool to scrape up to 10,000 URLs in a single job. Results are retrieved later via get_batch_results.

How does website crawling work?

Crawling is asynchronous: call create_crawl to start a crawl and get a crawl_id, then poll get_crawl_results to retrieve the scraped pages.

What authentication method is used?

Authentication uses a Bearer token in the Authorization header (for the remote endpoint) or the OLOSTEP_API_KEY environment variable (for local/docker setups).

Frequently asked questions

How do I get an Olostep API key?

What output formats are supported?

Scraped content can be returned as HTML, Markdown, JSON, or Plain Text (configurable per request).

Can I scrape multiple URLs at once?

Yes, use the `batch_scrape_urls` tool to scrape up to 10,000 URLs in a single job. Results are retrieved later via `get_batch_results`.

How does website crawling work?

Crawling is asynchronous: call `create_crawl` to start a crawl and get a `crawl_id`, then poll `get_crawl_results` to retrieve the scraped pages.

What authentication method is used?

Authentication uses a Bearer token in the `Authorization` header (for the remote endpoint) or the `OLOSTEP_API_KEY` environment variable (for local/docker setups).