Skip to content

About

No description, website, or topics provided.

Resources

Readme
MIT license
Activity
Custom properties

Stars

0 stars

Watchers

2 watching

Forks

0 forks
Report repository
BranchesTags

Repository files navigation

ScrapeOps MCP Server

A Model Context Protocol (MCP) server that exposes the full capabilities of the ScrapeOps Proxy API to LLM clients (Cursor, Claude Desktop, VS Code). Enable AI agents to browse the web, bypass anti-bots, render JavaScript, take screenshots, and perform structured data extraction autonomously.

Features

  • 🌐 Web Browsing - Browse any webpage with proxy support
  • 🌍 Geo-Targeting - Access websites from 12+ countries
  • 🏠 Residential/Mobile Proxies - Higher success rates on challenging sites
  • 🤖 Anti-Bot Bypass - Multiple bypass levels (Cloudflare, DataDome, PerimeterX, etc.)
  • 🖼️ Screenshots - Capture visual snapshots of pages
  • ⚡ JavaScript Rendering - Full headless browser capabilities
  • 📊 Structured Extraction - LLM-powered data extraction with schemas
  • 🔄 Auto-Extract - Domain-specific parsers for popular sites

Installation

Running with npx

env SCRAPEOPS_API_KEY=YOUR_API_KEY npx -y @scrapeops/mcp

Manual Installation

npm install -g @scrapeops/mcp

Configuration

Environment Variables

Variable Required Description
SCRAPEOPS_API_KEY Yes Your ScrapeOps API key from scrapeops.io

How It Works

The MCP server uses a simple, single-request approach:

  1. Basic Request: If no options are specified, a basic request is made (URL only)
  2. User-Specified Options: If options like render_js, residential, or premium are specified, they are used directly
  3. On Failure: Returns helpful error with suggestions - the AI/user decides what to do next
  4. No Auto-Retry: The server does not automatically retry with different options - this gives you full control

Example Flow:

  • You ask: "Scrape https://example.com"
  • Server makes basic request (no extra params)
  • If it fails (403), returns error with suggestions: "Try with residential: true"
  • You decide: "OK, scrape it with residential proxy"
  • Server makes request with residential: true

Running on Cursor

  1. Open Cursor Settings
  2. Go to Features > MCP Servers
  3. Click "+ Add new global MCP server"
  4. Enter the following configuration:
{
  "mcpServers": {
    "@scrapeops/mcp": {
      "command": "npx",
      "args": ["-y", "@scrapeops/mcp"],
      "env": {
        "SCRAPEOPS_API_KEY": "YOUR-API-KEY"
      }
    }
  }
}

Running on Claude Desktop

Add this to your claude_desktop_config.json:

{
  "mcpServers": {
    "@scrapeops/mcp": {
      "command": "npx",
      "args": ["-y", "@scrapeops/mcp"],
      "env": {
        "SCRAPEOPS_API_KEY": "YOUR_API_KEY_HERE"
      }
    }
  }
}

Running on VS Code

Add to your User Settings (JSON) via Ctrl + Shift + P → Preferences: Open User Settings (JSON):

{
  "mcp": {
    "inputs": [
      {
        "type": "promptString",
        "id": "apiKey",
        "description": "ScrapeOps API Key",
        "password": true
      }
    ],
    "servers": {
      "scrapeops": {
        "command": "npx",
        "args": ["-y", "@scrapeops/mcp"],
        "env": {
          "SCRAPEOPS_API_KEY": "${input:apiKey}"
        }
      }
    }
  }
}

Running on Windsurf

Add to your ./codeium/windsurf/model_config.json:

{
  "mcpServers": {
    "@scrapeops/mcp": {
      "command": "npx",
      "args": ["-y", "@scrapeops/mcp"],
      "env": {
        "SCRAPEOPS_API_KEY": "YOUR_API_KEY"
      }
    }
  }
}

Running a local HTTP server

You can run the same MCP server over HTTP instead of stdio. This is useful for development. It is not the hosted OAuth server.

1. Start the server:

# Set the port for HTTP/SSE mode (required for local server)
export PORT=8080
export SCRAPEOPS_API_KEY=your-api-key-here

# Run the server
npm start
# or if you have the package installed globally
scrapeops-mcp

The server exposes Streamable HTTP at http://localhost:8080/mcp (or the port specified by PORT). The legacy SSE endpoint remains available at /sse.

Note: If PORT is not set, the server runs in stdio mode (for use with npx in MCP clients like Cursor). Set PORT explicitly to run as an HTTP server.

2. Configure Cursor to connect to the local server:

Edit your Cursor MCP configuration file (typically at ~/.cursor/mcp.json or in Cursor Settings):

{
  "mcpServers": {
    "@scrapeops/mcp": {
      "url": "http://localhost:8080/mcp",
      "headers": {
        "scrapeops-api-key": "your-api-key-here"
      }
    }
  }
}

Note: When using the local HTTP transport, you can pass the API key either:

  • Via the scrapeops-api-key header in the configuration (as shown above), or
  • Via the SCRAPEOPS_API_KEY environment variable when starting the server

This local HTTP mode is separate from the hosted remote MCP. It keeps using your API key directly and does not require OAuth.

Remote MCP

The hosted server is for clients that connect over HTTPS instead of running the npm package, including Claude's remote connector directory.

Endpoint: https://mcp.scrapeops.io/mcp

Claude users add that URL as a remote MCP connector. The server responds to an unauthenticated request with a 401 and a WWW-Authenticate challenge. Claude then:

  1. Reads the OAuth protected-resource and authorization-server metadata
  2. Registers itself with dynamic client registration
  3. Sends the user to the ScrapeOps sign-in and approval page
  4. Calls the MCP endpoint with Authorization: Bearer <token>

The access token is resolved on the server to that user's ScrapeOps account. The account API key is used for Proxy API calls and is not returned to Claude. Tokens expire, refresh tokens rotate, and reuse of an old refresh token revokes that connection. A user can also revoke every connection at https://mcp.scrapeops.io/oauth/revoke-access.

npx -y @scrapeops/mcp with SCRAPEOPS_API_KEY is unchanged.

Deployment

The remote server is a separate Node process from the ScrapeOps frontend. Terminate TLS at the load balancer and forward https://mcp.scrapeops.io to the container. The process expects X-Forwarded-Proto: https in production.

docker build -t scrapeops-mcp .
docker run --rm -p 8080:8080 \
  -e PUBLIC_BASE_URL=https://mcp.scrapeops.io \
  -e TOKEN_ENCRYPTION_KEY="$(openssl rand -hex 32)" \
  -e SENTRY_DSN=https://[email protected]/0 \
  -v scrapeops-mcp-data:/data \
  scrapeops-mcp
Variable Required in production Description
PUBLIC_BASE_URL Yes Public origin, for example https://mcp.scrapeops.io
TOKEN_ENCRYPTION_KEY Yes 32-byte key (openssl rand -hex 32) used to encrypt account API keys at rest
REQUIRE_OAUTH Yes true so a process-wide API key cannot authenticate remote users
OAUTH_ENABLED Yes true
DATA_DIR No SQLite directory for OAuth clients and tokens. Defaults to ./data
SCRAPEOPS_BACKEND_URL No Login API. Defaults to https://backend.scrapeops.io/v1
SENTRY_DSN No Sends server errors to Sentry
LOG_FORMAT No Set to json for production logs
ALLOWED_ORIGINS No Extra browser origins, comma-separated. https://claude.ai and https://claude.com are already allowed

GET /health returns { "status": "ok" }. MCP and OAuth routes require HTTPS when NODE_ENV=production. Requests are rate limited per IP and per account.

Before submitting the Claude connector, confirm with a deployed server:

  1. https://mcp.scrapeops.io/mcp is accepted as a remote MCP URL.
  2. Claude discovers maps_web, extract_data, and return_links.
  3. Connect opens the ScrapeOps sign-in page.
  4. After approval, a tool call is billed to the signed-in account.
  5. Revoking access from Claude, or at /oauth/revoke-access, makes the next call return 401.

Available Tools

Tool 1: maps_web

General-purpose web browsing tool for reading pages, taking screenshots, and bypassing anti-bot protections.

Usage Examples:

// Simple page browse
{
  "name": "maps_web",
  "arguments": {
    "url": "https://example.com"
  }
}

// Screenshot from Germany with residential proxy
{
  "name": "maps_web",
  "arguments": {
    "url": "https://example.de",
    "country": "de",
    "residential": true,
    "screenshot": true
  }
}

// Bypass Cloudflare protection
{
  "name": "maps_web",
  "arguments": {
    "url": "https://protected-site.com",
    "bypass_level": "cloudflare_level_2",
    "residential": true,
    "render_js": true
  }
}

Tool 2: extract_data

Structured data extraction using auto-parsing or LLM-powered extraction.

Usage Examples:

// Auto-extract from known domain
{
  "name": "extract_data",
  "arguments": {
    "url": "https://www.amazon.com/dp/B09V3KXJPB",
    "mode": "auto"
  }
}

// LLM extraction for product page
{
  "name": "extract_data",
  "arguments": {
    "url": "https://shop.example.com/product/123",
    "mode": "llm",
    "data_schema": "product_page",
    "response_format": "json"
  }
}

// Extract job listings with anti-bot bypass
{
  "name": "extract_data",
  "arguments": {
    "url": "https://careers.example.com/jobs",
    "mode": "llm",
    "data_schema": "job_search_page",
    "bypass_level": "generic_level_2",
    "render_js": true
  }
}

User Stories

The Visual Debugger

"User complains a site looks broken in Germany. The AI calls maps_web(url='...', country='de', screenshot=true). The user sees the actual screenshot of the site rendered via a German residential IP."

The Efficient Scraper

"User needs pricing data. Instead of fetching HTML and parsing it (wasting tokens), the AI calls extract_data(url='...', mode='llm', data_schema='product_page'). ScrapeOps handles the heavy lifting, and the AI just displays the final JSON."

The Bypass Expert

"The AI tries to access a site and gets blocked. It automatically retries the request using maps_web with bypass_level='generic_level_3' and residential=true to overcome the blockage."

System Configuration

The server includes configurable retry parameters with exponential backoff:

const RETRY_CONFIG = {
  maxAttempts: 1,      
  initialDelay: 1000, 
};

Retry Behavior:

  • Network errors are retried once regardless of maxAttempts setting
  • To enable retries, set SCRAPEOPS_RETRY_MAX_ATTEMPTS environment variable

Custom Configuration Example:

# Enable retries with 3 attempts
export SCRAPEOPS_RETRY_MAX_ATTEMPTS=3
export SCRAPEOPS_RETRY_INITIAL_DELAY=1000

Error Handling

Status Code Error Resolution
401 Invalid API Key Check your SCRAPEOPS_API_KEY environment variable
403 Forbidden Target website blocking request - consider using advanced parameters
404 Not Found Verify the URL is correct
429 Rate Limited Too many requests - wait before retrying (NOT auto-retried)
500 Server Error Automatically retried up to 3 times with exponential backoff
502/503 Gateway/Service Error Temporary issue - NOT auto-retried

Development

# Install dependencies
npm install

# Build
npm run build

# Run locally (stdio transport)
SCRAPEOPS_API_KEY=your-key npm start

# Run tests
npm test

API Reference

Base URL: https://proxy.scrapeops.io/v1/

Authentication: Query parameter ?api_key=... (managed via server-side environment variables)

For full API documentation, visit ScrapeOps Documentation.

License

MIT License - see LICENSE file for details.

Support

About

No description, website, or topics provided.

Resources

Readme
MIT license
Activity
Custom properties

Stars

0 stars

Watchers

2 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages