Selenium integration

Connect Selenium to a Crawlhub proxy.

Run Selenium through a local HTTP bridge that authenticates to your Crawlhub proxy.

Crawlhub.ai and Selenium integration guide. Proxy routing for WebDriver browser sessions. — Selenium integration guide. View full size

Browser routing and authentication

Selenium sets the browser’s proxy route. A small local bridge handles the upstream username and password, so your browser does not need an authentication extension or a handler for every website challenge.

Local browser endpoint

The bridge listens on 127.0.0.1 and forwards to your assigned Crawlhub proxy.

Upstream authentication

The Node.js bridge supplies the proxy login. Selenium only receives the local address.

Standard Selenium code

Keep your existing Chrome options, locators and waits.

Two processes, clear lifetime

Keep the bridge running while the browser works. Stop it when the job finishes.

Setup guide

Set up Selenium with Crawlhub

Start with one connection check. Add your workload after the route works.

Selenium

Selenium connects to the local bridge

proxy.http_proxy = local_host_and_port
proxy.ssl_proxy = local_host_and_port
options = webdriver.ChromeOptions()
options.proxy = proxy
driver = webdriver.Chrome(options=options)

Before you start

An active Crawlhub proxy and its assigned host, port, username and password. PROXY_SERVER includes the protocol and port, but no username or password. Use the values from your account; these examples do not activate a proxy.

Python with Selenium 4, Node.js supported by proxy-chain, and compatible Chrome/ChromeDriver on the same machine. This guide uses a local HTTP bridge with an HTTP(S) upstream proxy.

Install Selenium and Chrome
Create a Python environment and install Selenium. Install Chrome and a compatible driver before the connection check. Selenium Manager can obtain the driver when network access is available.
Install proxy-chain in the same project folder as selenium_bridge.mjs. The bridge and browser should run on the same machine.
Run in your terminal
```
python3 -m venv .venv
source .venv/bin/activate
python -m pip install selenium
npm init -y
npm install proxy-chain
```

Start the authenticated bridge

Save both helper files below in the same project folder. The CLI wrapper imports proxy_bridge.mjs.

Set your assigned proxy values in the first terminal, then save and run selenium_bridge.mjs. It prints a local URL such as http://127.0.0.1:49152. The port is selected automatically.

Leave that terminal running. The bridge binds only to loopback and prints no upstream credentials. Stop it with Ctrl+C when you are done.

Run in your terminal

export PROXY_SERVER='http://YOUR_PROXY_HOST:YOUR_PORT'
export PROXY_USERNAME='YOUR_PROXY_USERNAME'
export PROXY_PASSWORD='YOUR_PROXY_PASSWORD'
node selenium_bridge.mjs

Download proxy_bridge.mjs

proxy_bridge.mjs

import { Server } from 'proxy-chain';

export async function createProxyBridge() {
  const { PROXY_SERVER, PROXY_USERNAME, PROXY_PASSWORD } = process.env;
  if (!PROXY_SERVER || !PROXY_USERNAME || !PROXY_PASSWORD) {
    throw new Error('Set PROXY_SERVER, PROXY_USERNAME and PROXY_PASSWORD.');
  }
  const upstream = new URL(PROXY_SERVER);
  if (!['http:', 'https:'].includes(upstream.protocol) ||
      upstream.username || upstream.password || upstream.search ||
      upstream.hash || (upstream.pathname && upstream.pathname !== '/')) {
    throw new Error('Use an HTTP(S) proxy server URL without credentials, a path or a query.');
  }
  upstream.username = PROXY_USERNAME;
  upstream.password = PROXY_PASSWORD;
  const server = new Server({
    host: '127.0.0.1',
    port: 0,
    prepareRequestFunction: () => ({ upstreamProxyUrl: upstream.href }),
  });
  await server.listen();
  return {
    url: `http://127.0.0.1:${server.port}`,
    close: () => server.close(true),
  };
}

Download selenium_bridge.mjs

selenium_bridge.mjs

import { createProxyBridge } from './proxy_bridge.mjs';

const bridge = await createProxyBridge();
console.log(bridge.url);
for (const signal of ['SIGINT', 'SIGTERM']) {
  process.once(signal, async () => { await bridge.close(); process.exit(0); });
}

Run Selenium through the local URL

Open a second terminal, activate the Python environment, and save selenium_proxy.py in the project folder. Set LOCAL_PROXY_SERVER to the exact URL printed by the bridge, not the example port shown here.

The script routes both HTTP and HTTPS browser traffic through the bridge, sets a page-load timeout and closes the driver after printing the title.

Run in your terminal

source .venv/bin/activate
export LOCAL_PROXY_SERVER='http://127.0.0.1:PORT_PRINTED_BY_BRIDGE'
export TARGET_URL='https://example.com/'
python selenium_proxy.py

Download selenium_proxy.py

selenium_proxy.py

import os
from urllib.parse import urlsplit
from selenium import webdriver
from selenium.webdriver.common.proxy import Proxy, ProxyType

server = urlsplit(os.environ['LOCAL_PROXY_SERVER'])
if server.scheme != 'http' or server.hostname != '127.0.0.1' or not server.port:
    raise ValueError('Set LOCAL_PROXY_SERVER to the URL printed by the local bridge.')
proxy = Proxy()
proxy.proxy_type = ProxyType.MANUAL
proxy.http_proxy = server.netloc
proxy.ssl_proxy = server.netloc
options = webdriver.ChromeOptions()
options.proxy = proxy
options.add_argument('--headless=new')
driver = webdriver.Chrome(options=options)
try:
    driver.set_page_load_timeout(30)
    driver.get(os.environ.get('TARGET_URL', 'https://example.com/'))
    print(driver.title)
finally:
    driver.quit()

Verify the session
A successful request confirms that the page loaded, but it does not by itself prove the exit IP. Run the same code against an IP-check endpoint you trust, inspect the returned address, then switch to your target. Keep the requested location and session settings consistent when comparing results.
Keep the browser and assigned sticky proxy session together for a related workflow. Use Selenium waits for the specific content you need instead of assuming that page navigation means all asynchronous content is ready.

When the connection fails

Use the error to narrow down the next check.

Connection refused at localhost

Keep the bridge running and copy its printed port exactly. Restarting the bridge may choose a new port.

Upstream 407

Check the three proxy environment variables in the bridge terminal, then restart the bridge after correcting them.

Driver cannot start

Resolve Chrome and driver installation first. For Selenium Grid or a remote browser, the bridge must be reachable from the browser host; this loopback example is for a local browser.

Start free