Rotating Proxies in Colly: A Framework-Idiomatic Guide for Go Scrapers

Rotating proxies in Colly is the single most effective way to keep a Go web scraper alive against rate-limited targets. Colly—Go's most popular scraping framework—ships with a built-in proxy switcher, an async collector model, and goquery-backed DOM traversal that together let you distribute requests across hundreds of IPs with a few lines of code. This guide walks through the idiomatic proxy surface, shows runnable examples against gate.proxyhat.com:8080, and covers production pitfalls that catch teams at scale.

Legal note: Scraping public data is generally permissible in many jurisdictions, but the U.S. Computer Fraud and Abuse Act (CFAA) and the EU's GDPR both have boundaries—accessing non-public pages, bypassing authentication, or processing personal data without a lawful basis can expose you to liability. Always respect robots.txt, target terms of service, and rate limits. See 18 U.S.C. § 1030 and GDPR.eu for authoritative summaries.

Colly's Collector Model: Where Proxies Hook In

Colly is built around the colly.Collector—a struct that manages an HTTP client, a request queue, callback registration, and storage backends. Understanding its lifecycle is essential before layering proxy rotation on top.

The collector exposes three callback families that matter for proxy work:

• OnRequest — fires before each request is sent. This is where you can inspect or mutate headers, inject per-request proxy logic, or short-circuit a request entirely with r.Abort().
• OnHTML — fires after the response body is parsed via goquery, giving you jQuery-like selectors (e.DOM.Find(".price")) to extract data.
• OnError — fires when a request fails (network error, non-2xx status, or a proxy timeout). This is your retry hook.

By default, Colly runs synchronously—c.Visit(url) blocks until the response is processed. Enabling async mode with c.Async = true turns the collector into a concurrent pipeline: requests are dispatched in goroutines, bounded by c.Limit() rules, and you call c.Wait() to block until the queue drains. Async mode is where proxy rotation earns its keep—a single IP making 200 concurrent requests is a ban magnet; 200 IPs making one request each is invisible.

Under the hood, Colly uses Go's net/http transport, which means any http.RoundTripper customization—custom TLS configs, dial timeouts, keep-alive tuning—is available via c.WithTransport(). The proxy selection logic plugs in one layer above the transport, through c.SetProxyFunc().

The Idiomatic Proxy Surface: RoundRobinProxySwitcher and SetProxyFunc

Colly provides two native mechanisms for proxy rotation. The first is proxy.RoundRobinProxySwitcher, a ready-made switcher that cycles through a list of proxy URLs in order:

import "github.com/gocolly/colly/v2/proxy"

switcher, err := proxy.RoundRobinProxySwitcher(
    "http://user-country-US:pass@gate.proxyhat.com:8080",
    "http://user-country-DE:pass@gate.proxyhat.com:8080",
    "http://user-country-GB:pass@gate.proxyhat.com:8080",
)
if err != nil {
    log.Fatal(err)
}
c.SetProxyFunc(switcher)

The switcher returns a func(*http.Request) (*url.URL, error)—exactly the signature that SetProxyFunc expects. Each call to the function advances the round-robin index and returns the next proxy URL. This is simple and correct for small endpoint lists, but it has a limitation: the list is fixed at construction time. For dynamic rotation—where you want to vary the geo-targeting or session ID per request—you need a custom switcher.

The second mechanism is passing any function with the signature func(*http.Request) (*url.URL, error) directly to SetProxyFunc. This is the idiomatic extension point. Colly calls it before every request, giving you the *http.Request object so you can make per-request decisions based on the target URL, a counter, or a random selection:

c.SetProxyFunc(func(r *http.Request) (*url.URL, error) {
    // Rotate country and session per request
    country := pickRandomCountry()  // e.g. "US", "DE", "GB"
    session := generateSessionID()  // e.g. "sess-9f2a1b"
    proxyStr := fmt.Sprintf(
        "http://user-country-%s-session-%s:pass@gate.proxyhat.com:8080",
        country, session,
    )
    return url.Parse(proxyStr)
})

Both http:// and socks5:// schemes are supported. For SOCKS5, point at port 1080 instead of 8080:

socks5://user-country-DE-session-abc123:pass@gate.proxyhat.com:1080

The url.Parse call handles credential embedding correctly—Colly passes the parsed URL to Go's transport, which injects Proxy-Authorization headers automatically. No manual header manipulation needed.

Why Residential IPs Are Required for Hard Targets

Datacenter IP ranges are published in public blocklists that anti-bot services like Cloudflare, DataDome, and PerimeterX query in real time. A request from an AWS us-east-1 block is flagged instantly, regardless of headers or behavior. Residential proxies—IPs assigned by ISPs to real households—don't appear in datacenter ASN databases, making them the baseline requirement for scraping targets with modern bot protection.

ProxyHat's residential endpoints accept geo-targeting and session flags in the username, which means you can control two orthogonal rotation axes without maintaining a pool of proxy URLs:

• Geo-rotation — user-country-DE-city-berlin routes through a Berlin residential IP. Rotating countries per request distributes your footprint and avoids per-country rate caps.
• Session stickiness — user-session-abc123 pins all requests with that session ID to the same exit IP. This is critical for multi-page flows (login → browse → checkout) where the target ties your session to an IP.

The combination is powerful: use a fresh session ID per logical scraping job for IP stickiness within that job, but rotate the country across jobs to avoid geographic clustering. A target seeing 500 requests from Berlin in 10 seconds is suspicious; 500 requests spread across 20 German cities looks like normal traffic.

Proxy Type Comparison

For most production scraping workloads, residential proxies hit the sweet spot. Explore ProxyHat's residential locations to see available geo-targeting options.

A Runnable Go Example: Residential RoundRobin with Limit Rules

Here's a complete, runnable scraper that combines a residential proxy switcher with Colly's rate-limiting and error handling. It scrapes a product listing page, extracts prices, and retries failed requests on a cloned collector.

package main

import (
    "fmt"
    "log"
    "math/rand"
    "net/url"
    "time"

    "github.com/gocolly/colly/v2"
)

var countries = []string{"US", "DE", "GB", "FR", "CA", "AU"}

func randomProxy() func(*http.Request) (*url.URL, error) {
    return func(r *http.Request) (*url.URL, error) {
        country := countries[rand.Intn(len(countries))]
        session := fmt.Sprintf("sess-%d", time.Now().UnixNano())
        proxyStr := fmt.Sprintf(
            "http://user-country-%s-session-%s:YOUR_PASSWORD@gate.proxyhat.com:8080",
            country, session,
        )
        return url.Parse(proxyStr)
    }
}

func main() {
    c := colly.NewCollector(
        colly.UserAgent("Mozilla/5.0 (compatible; ScraperBot/1.0)"),
    )

    // Async mode + rate limiting
    c.Async = true
    err := c.Limit(&colly.LimitRule{
        DomainRegexp: "example-shop\.com",
        Delay:       2 * time.Second,
        RandomDelay: 1 * time.Second,
        Parallelism: 5,
    })
    if err != nil {
        log.Fatal(err)
    }

    // Set the residential proxy switcher
    c.SetProxyFunc(randomProxy())

    // Custom transport for TLS tuning
    c.WithTransport(&http.Transport{
        TLSHandshakeTimeout:   10 * time.Second,
        ResponseHeaderTimeout: 15 * time.Second,
        IdleConnTimeout:       90 * time.Second,
    })

    c.OnHTML(".product-card", func(e *colly.HTMLElement) {
        name := e.ChildText(".product-name")
        price := e.ChildText(".price")
        fmt.Printf("%s — %s\n", name, price)
    })

    c.OnError(func(r *colly.Response, err error) {
        log.Printf("Request failed: %s — %v (status %d)",
            r.Request.URL, err, r.StatusCode)
        // Retry on a clone with a fresh proxy
        if r.StatusCode == 403 || r.StatusCode == 429 {
            clone := r.Request.Colly.Clone()
            clone.SetProxyFunc(randomProxy())
            clone.Visit(r.Request.URL.String())
        }
    })

    c.Visit("https://example-shop.com/products")
    c.Wait()
}

Key things to notice: the LimitRule caps parallelism at 5 concurrent requests per domain with a 2-second base delay plus up to 1 second of jitter—this keeps you under most rate-limit thresholds. The OnError handler clones the collector (which inherits all callbacks) and retries with a fresh proxy, which is the idiomatic Colly pattern for transient failures.

For more scraping use cases, see ProxyHat's web scraping overview and SERP tracking use case.

Production Patterns: Retries, Delays, TLS, and Distributed Storage

Retries with c.Clone()

Colly's Clone() method creates a shallow copy of a collector with the same callbacks but a fresh HTTP client. This is the recommended way to retry: clone, swap the proxy function, and re-visit. Avoid retrying on the same collector instance if you've already called Wait()—the internal wait group won't reset cleanly.

c.OnError(func(r *colly.Response, err error) {
    if r.StatusCode >= 500 || r.StatusCode == 429 {
        time.Sleep(5 * time.Second)  // backoff
        retry := c.Clone()
        retry.SetProxyFunc(randomProxy())
        retry.Visit(r.Request.URL.String())
        retry.Wait()
    }
})

RandomDelay and Parallelism Tuning

The RandomDelay field adds jitter on top of the base Delay, which is critical for avoiding detection. A fixed 2-second delay produces a suspiciously regular request pattern; adding 0–1 seconds of randomness makes the traffic look human. Set Parallelism conservatively—5 to 10 concurrent requests per domain is a safe starting point for residential proxies. Pushing to 50+ concurrent sessions risks triggering behavioral analysis even with clean IPs.

Custom Transport TLS Configuration

Some targets reject connections with default Go TLS fingerprints. While Colly can't fully spoof a browser's TLS ClientHello (that requires utls or a browser engine), you can at least tune timeouts and cipher suites:

c.WithTransport(&http.Transport{
    TLSHandshakeTimeout:   10 * time.Second,
    ResponseHeaderTimeout: 15 * time.Second,
    ExpectContinueTimeout: 1 * time.Second,
    MaxIdleConns:          100,
    MaxIdleConnsPerHost:   10,
    IdleConnTimeout:       90 * time.Second,
})

For targets with aggressive TLS fingerprinting, consider pairing Colly with a headless browser fleet for the protected pages and using Colly for the unprotected API endpoints.

Distributed Scraping via Redis Storage

When you scale beyond a single process, Colly's storage backends let you share visited-URL tracking and cookies across multiple worker instances. The Redis backend is the production choice:

import "github.com/gocolly/redisstorage"

storage := &redisstorage.Storage{
    Address:  "redis-cluster.internal:6379",
    Password: "",
    Prefix:   "scraper_job_42",
    Client:   nil,
}
err := c.SetStorage(storage)
if err != nil {
    log.Fatal(err)
}
defer storage.Close()

With Redis storage, multiple Colly instances running in containers (Kubernetes pods, ECS tasks, etc.) can coordinate: each worker pulls URLs from a shared queue, marks them as visited in Redis, and avoids duplicate fetches. This is the standard pattern for horizontally scaling a Colly scraper—run N containers, each with its own proxy switcher, all pointing at the same Redis cluster. A fleet of 10 containers with 5 concurrent requests each gives you 50 concurrent residential sessions, which is enough throughput for most e-commerce price monitoring workloads.

See the Colly package documentation on pkg.go.dev for the full storage backend API.

When NOT to Use Colly (and What to Reach For Instead)

Colly is an HTTP client with HTML parsing—it does not execute JavaScript. If your target renders content via a client-side framework (React, Vue, Svelte), Colly will see an empty <div id="root"> and nothing else. In those cases, you need a browser-based tool:

• Chromedp — Go's native Chrome DevTools Protocol client. Pairs well with Colly: use Chromedp for the JS-heavy landing page, extract the API endpoint the page calls, then hand off to Colly for bulk fetching.
• Playwright / Puppeteer — Node.js or Python, with full browser automation. Slower but handles any SPA.
• Rod — Another Go option, simpler API than Chromedp for some use cases.

The hybrid pattern is common in production: use a headless browser fleet (with its own proxy rotation) to handle the initial page load and extract API URLs or authentication tokens, then feed those to a Colly fleet for the high-throughput data extraction. This gives you browser-grade evasion where you need it and HTTP-grade speed where you don't.

Key Takeaways

• Use SetProxyFunc with a custom closure for dynamic rotation—vary country and session per request by embedding flags in the ProxyHat username.
• Residential proxies are the baseline for targets with modern anti-bot. Datacenter IPs are detected via ASN blocklists in real time.
• Always enable async mode + Limit() — unbounded concurrency gets you banned even with clean IPs. Start at 5 parallel requests with 2–3 seconds of randomized delay.
• Retry with c.Clone() on 429/503 responses, with exponential backoff and a fresh proxy.
• Scale horizontally with Redis storage — multiple containerized Colly workers sharing a Redis backend gives you linear throughput scaling.
• Colly can't render JavaScript. For SPAs, use a browser engine (Chromedp, Rod, Playwright) for the protected pages and Colly for the API layer.
• Stay legal. Scrape public data only, respect robots.txt, and consult GDPR guidance if you're processing any personal data from EU residents.

Ready to put this into practice? Check out ProxyHat's pricing plans for residential proxy access, or read the ProxyHat documentation for full connection details.

Frequently Asked Questions

What is rotating proxies in Colly?

Rotating proxies in Colly means using the framework's SetProxyFunc method to assign a different proxy IP to each outgoing HTTP request. Colly provides proxy.RoundRobinProxySwitcher for simple fixed-list rotation, or you can pass a custom func(*http.Request) (*url.URL, error) that dynamically selects a proxy per request—varying country, city, or session ID through the ProxyHat username flags.

Why does rotating proxies in Colly matter for proxy users?

Without rotation, all requests originate from a single IP, making rate-limiting and IP bans trivial for anti-bot systems. Rotation distributes requests across many residential IPs, keeping per-IP request counts low enough to stay under detection thresholds. For high-throughput scraping—price monitoring, SERP tracking, or data collection at scale—rotation is the difference between a scraper that runs for hours and one that dies in minutes.

Which proxy type works best for rotating proxies in Colly?

Residential proxies are the best default for Colly scrapers targeting protected sites. They use ISP-assigned IPs that don't appear in datacenter ASN blocklists, so they pass the first line of anti-bot checks. Datacenter proxies are fine for unprotected APIs or internal testing but are flagged instantly by Cloudflare, DataDome, and similar services. Mobile proxies offer the highest evasion but at a premium cost—reserve them for the hardest targets.

How do you avoid blocks when implementing rotating proxies in Colly?

Combine four practices: (1) use residential proxies with per-request country and session rotation via ProxyHat's username flags; (2) set Limit() rules with a 2–3 second base delay plus random jitter and cap parallelism at 5–10 per domain; (3) retry 429/503 responses on a cloned collector with a fresh proxy and exponential backoff; (4) set a realistic User-Agent and avoid making requests in a perfectly regular pattern.