Scrapy integration

Route Scrapy requests through Crawlhub.

Add an HTTP proxy to each request and keep Scrapy in charge of crawling, parsing and output.

Crawlhub.ai and Scrapy integration guide. Proxy connections for your Scrapy spiders. — Scrapy integration guide. View full size

A proxy for the downloader

Scrapy sends requests through its downloader middleware. HttpProxyMiddleware reads the proxy URL from request metadata and handles HTTP proxy authentication.

Proxy every downloader request

A small middleware assigns the proxy to the page, robots.txt and callback-generated requests.

Encoded credentials

URL-encode the username and password before building an authenticated proxy URL.

Normal spider output

Keep selectors, item fields and export formats in your spider.

Verified HTTPS responses

The example enables server certificate verification and bounds concurrency, download time and retries.

Setup guide

Set up Scrapy with Crawlhub

Start with one connection check. Add your workload after the route works.

Scrapy

Proxy on the request

def process_request(self, request):
    request.meta['proxy'] = self.proxy

Before you start

An active proxy connection with its assigned protocol, hostname, port, full username and password. Copy those fields from your account or activation details. A browser profile does not create proxy credentials.

Python 3.10 or newer and Scrapy 2.20 or newer. This recipe uses an HTTP proxy endpoint for HTTP and HTTPS destination pages. TLS-to-proxy and SOCKS connections require a compatible download handler and are outside this example.

Install Scrapy in a virtual environment
Keep the spider dependencies separate from your system Python. Activate the environment before installing or running the spider.
Run in your terminal
```
python3 -m venv .venv
source .venv/bin/activate
python -m pip install 'scrapy>=2.20'
```
Set your connection details
Use an http://HOST:PORT proxy server URL for this recipe, even when TARGET_URL uses https://. Supply the raw username and password; the middleware URL-encodes them when constructing the proxy URL.
Run in your terminal
```
export PROXY_SERVER='http://YOUR_PROXY_HOST:YOUR_PORT'
export PROXY_USERNAME='YOUR_PROXY_USERNAME'
export PROXY_PASSWORD='YOUR_PROXY_PASSWORD'
export TARGET_URL='https://example.com/'
```

Run the connection-check spider

Save scrapy_proxy.py and run it without creating a full Scrapy project. The spider respects robots.txt and exports the page URL, status and title.

The spider explicitly enables DOWNLOAD_VERIFY_CERTIFICATES. Keep certificate verification enabled for HTTPS destinations.

Run in your terminal

scrapy runspider scrapy_proxy.py -O results.json

Download scrapy_proxy.py

scrapy_proxy.py

import os
from urllib.parse import quote, urlsplit, urlunsplit
import scrapy

class CrawlhubProxyMiddleware:
    def __init__(self):
        server = urlsplit(os.environ['PROXY_SERVER'])
        if server.scheme != 'http' or not server.hostname or server.username or server.password or server.query or server.fragment or server.path not in ('', '/'):
            raise ValueError('Use an HTTP proxy server URL without credentials, a path or a query.')
        user = quote(os.environ['PROXY_USERNAME'], safe='')
        password = quote(os.environ['PROXY_PASSWORD'], safe='')
        self.proxy = urlunsplit((server.scheme, f'{user}:{password}@{server.netloc}', '', '', ''))

    def process_request(self, request):
        # Include robots.txt and requests generated by callbacks.
        request.meta['proxy'] = self.proxy

class ProxyCheckSpider(scrapy.Spider):
    name = 'crawlhub_proxy_check'
    custom_settings = {
        'DOWNLOADER_MIDDLEWARES': {CrawlhubProxyMiddleware: 740},
        'CONCURRENT_REQUESTS': 1,
        'DOWNLOAD_TIMEOUT': 30,
        'DOWNLOAD_VERIFY_CERTIFICATES': True,
        'RETRY_TIMES': 1,
        'ROBOTSTXT_OBEY': True,
    }

    async def start(self):
        target = os.environ.get('TARGET_URL', 'https://example.com/')
        yield scrapy.Request(target, callback=self.parse)

    def parse(self, response):
        yield {
            'url': response.url,
            'status': response.status,
            'title': response.css('title::text').get(),
        }

Expand the spider carefully
Inspect results.json before adding pagination or more URLs. The supplied downloader middleware applies the proxy consistently, including robots.txt. Keep that middleware enabled when moving the spider into a larger project.
A successful request confirms that the page loaded, but it does not by itself prove the exit IP. Run the same code against an IP-check endpoint you trust, inspect the returned address, then switch to your target. Keep the requested location and session settings consistent when comparing results.
If a destination disallows the request in robots.txt, use a different allowed test destination instead of treating that result as a proxy failure.

When the connection fails

Use the error to narrow down the next check.

407 from the downloader

Confirm the proxy URL and credential encoding. Review sanitized logs without exposing the authenticated proxy URL.

Some requests bypass the proxy

Keep the proxy middleware enabled before HttpProxyMiddleware. A proxy set only on the first page request can leave robots.txt or later requests on a direct route.

No items exported

Check robots.txt decisions, HTTP statuses and selectors. Successful transport does not guarantee a matching title element.

Start free