Rotating Proxies in Colly: A Framework-Idiomatic Guide for Go Developers

If you've built a Go scraper with Colly and watched it get IP-banned after 50 requests, you already know why rotating proxies in Colly is non-negotiable for production scraping. Anti-bot systems fingerprint your IP, throttle by subnet, and block datacenter ranges outright. Colly gives you a clean, idiomatic proxy surface—but most guides stop at a single hard-coded proxy string, which defeats the purpose entirely.

Before diving in: this guide covers scraping publicly accessible data only. In the U.S., the Computer Fraud and Abuse Act (CFAA) and in the EU, the GDPR set boundaries on what you can collect and how. Always respect robots.txt and target-site terms of service. See the official RFC 9308 for the robots.txt specification.

Why Rotating Proxies in Colly Matter

Colly is Go's most popular scraping framework, built on top of goquery for DOM traversal and the standard net/http stack for transport. It's fast—a single-threaded collector can handle 1,000+ requests per second on modest hardware—but that speed is exactly what gets you blocked. A single IP hammering a target at 200ms intervals triggers rate-limiting, CAPTCHA challenges, or outright 403 responses within minutes.

Rotating proxies solve this by distributing requests across many IP addresses. But not all proxies are equal. Datacenter IPs are cheap and fast but flagged by most modern anti-bot systems. Residential IPs—real ISP-assigned addresses—blend in with organic traffic and are dramatically harder to detect. For hard targets like search engines, ticketing platforms, or e-commerce price monitors, residential proxies are the baseline requirement, not a luxury.

Colly's Collector Model: Callbacks, Async, and goquery

Understanding Colly's architecture is essential before wiring in proxies. A Collector is Colly's central object. You register callbacks on it, and each callback fires at a specific point in the request lifecycle:

• OnRequest — fires before each HTTP request is sent. This is where you can inspect or modify headers, the proxy assignment, and the URL.
• OnResponse — fires after a response is received, before HTML parsing. Useful for inspecting status codes or raw body.
• OnHTML — fires when a CSS selector matches elements in the parsed DOM. Powered by goquery, this gives you jQuery-like traversal (e.ChildText(), e.ForEach(), e.Attr()).
• OnError — fires on network errors, non-2xx status codes, or callback panics. This is your retry hook.
• OnScraped — fires after all callbacks complete for a request. Good for cleanup or logging.

Colly runs asynchronously by default. When you call c.Visit(), the request is dispatched on a goroutine. The Limit() method controls concurrency per domain—setting Parallelism and Delay prevents you from overwhelming a single target. This matters for proxy rotation because each concurrent request may need a different proxy, and Colly's proxy switcher is called per-request, not per-collector.

The Idiomatic Proxy Surface: RoundRobinProxySwitcher and SetProxyFunc

Colly exposes two idiomatic mechanisms for proxy rotation. The first is proxy.RoundRobinProxySwitcher, a built-in round-robin switcher from the github.com/gocolly/colly/v2/proxy package:

package main

import (
    "log"
    "github.com/gocolly/colly/v2"
    "github.com/gocolly/colly/v2/proxy"
)

func main() {
    c := colly.NewCollector()

    proxies := []string{
        "http://user-country-US:YOUR_PASSWORD@gate.proxyhat.com:8080",
        "http://user-country-DE:YOUR_PASSWORD@gate.proxyhat.com:8080",
        "http://user-country-GB:YOUR_PASSWORD@gate.proxyhat.com:8080",
    }

    switcher, err := proxy.RoundRobinProxySwitcher(proxies...)
    if err != nil {
        log.Fatal(err)
    }
    c.SetProxyFunc(switcher)

    c.OnHTML("title", func(e *colly.HTMLElement) {
        log.Println(e.Text)
    })

    c.Visit("https://example.com")
}

This works for simple cases but has a limitation: the proxy list is fixed at initialization. For production scraping, you need dynamic rotation—different countries, sticky sessions for multi-page flows, or per-request geo-targeting. That's where the second mechanism shines: c.SetProxyFunc(func(*http.Request) (*url.URL, error)). You pass a custom function that receives each http.Request and returns a proxy URL. This is the framework-idiomatic extension point for any proxy logic.

Why Residential IPs Beat Datacenter for Hard Targets

Residential proxies route traffic through real ISP-assigned IP addresses. When a target site checks the ASN of an incoming request, it sees Comcast, Deutsche Telekom, or Vodafone—not AWS or DigitalOcean. This makes residential IPs far harder to block at the network level.

ProxyHat's residential proxy gateway supports geo-targeting and session stickiness through the username field. The format is:

• user-country-DE:pass — rotate within German IPs
• user-country-DE-city-berlin:pass — pin to Berlin
• user-session-abc123:pass — sticky session (same IP for the session ID)
• user-country-US-session-abc123:pass — sticky session within US IPs

Combining -country- and -session- flags gives you per-request geo-targeting with optional stickiness. For SERP tracking, you want a fresh IP per request. For multi-page checkout flows, you want the same IP across all requests in the flow.

For a deeper comparison of proxy types and their trade-offs, see the web scraping use case guide. You can also browse available proxy locations to plan your geo-targeting strategy.

Runnable Example: Residential Proxy Rotation with Geo-Targeting

Here's a complete, runnable Colly scraper with a custom proxy switcher that rotates country and session per request, enforces rate limits, and uses Colly's queue for distributed URL management:

package main

import (
    "fmt"
    "log"
    "net/http"
    "net/url"
    "sync/atomic"
    "time"

    "github.com/gocolly/colly/v2"
    "github.com/gocolly/colly/v2/queue"
)

var reqCounter uint64

func residentialSwitcher(r *http.Request) (*url.URL, error) {
    n := atomic.AddUint64(&reqCounter, 1)
    countries := []string{"US", "DE", "GB", "FR", "JP", "CA", "AU"}
    country := countries[n%uint64(len(countries))]
    session := fmt.Sprintf("rot-%d", n)

    proxyStr := fmt.Sprintf(
        "http://user-country-%s-session-%s:YOUR_PASSWORD@gate.proxyhat.com:8080",
        country, session,
    )
    return url.Parse(proxyStr)
}

func main() {
    c := colly.NewCollector(
        colly.MaxDepth(2),
        colly.UserAgent("Mozilla/5.0 (compatible; MyBot/1.0)"),
    )

    c.SetProxyFunc(residentialSwitcher)

    // Rate limit: 20 concurrent requests, 2s delay + 500ms jitter
    c.Limit(&colly.LimitRule{
        DomainGlob:  "*",
        Parallelism:  20,
        Delay:        2 * time.Second,
        RandomDelay:  500 * time.Millisecond,
    })

    c.OnHTML("a[href]", func(e *colly.HTMLElement) {
        link := e.Request.AbsoluteURL(e.Attr("href"))
        e.Request.Visit(link)
    })

    c.OnHTML("title", func(e *colly.HTMLElement) {
        fmt.Printf("[%s] %s\n", e.Request.URL, e.Text)
    })

    c.OnError(func(r *colly.Response, err error) {
        log.Printf("Error %d on %s: %v", r.StatusCode, r.Request.URL, err)
    })

    // In-memory queue for URL management
    q, _ := queue.New(
        2,
        &queue.InMemoryQueueStorage{MaxSize: 10000},
    )
    q.AddURL("https://example.com")
    q.Run(c)
}

The key design decisions here: atomic.AddUint64 gives us a thread-safe counter for unique session IDs. The LimitRule with Parallelism: 20 and a 2-second base delay plus 500ms jitter keeps request patterns looking organic. The queue decouples URL discovery from URL processing, which matters at scale.

Production Patterns: Retries, TLS, and Distributed Queues

Retries with c.Clone()

Colly's OnError callback is where you handle transient failures. For 429 (Too Many Requests) or 5xx errors, you can retry with backoff. Colly's Response.Request.Retry() method re-queues the request. For more complex retry logic—different proxy, different headers—use c.Clone() to create a shallow copy of the collector with the same callbacks but a fresh transport:

c.OnError(func(r *colly.Response, err error) {
    if r.StatusCode == 429 || r.StatusCode >= 500 {
        log.Printf("Retrying %s (status %d)", r.Request.URL, r.StatusCode)
        time.Sleep(5 * time.Second)
        r.Request.Retry(3)
    }
})

Custom TLS Configuration

Some targets require specific TLS settings—SNI matching, custom cipher suites, or certificate pinning bypass. Colly exposes the underlying http.Transport via c.WithTransport(). You can set a custom tls.Config to control TLS fingerprinting:

import (
    "crypto/tls"
    "net/http"
    "github.com/gocolly/colly/v2"
)

c := colly.NewCollector()
c.WithTransport(&http.Transport{
    TLSClientConfig: &tls.Config{
        InsecureSkipVerify: false,
        MinVersion:         tls.VersionTLS12,
    },
    Proxy: http.ProxyFromEnvironment,
})

Note that when using SetProxyFunc, Colly handles proxy assignment internally—you don't need to set Proxy on the transport manually. The SetProxyFunc approach is preferred because it integrates with Colly's per-request lifecycle.

Distributed Scraping with Redis Storage

For multi-instance scraping—running Colly across containers or VMs—you need a shared queue and deduplication state. Colly's queue package supports pluggable storage backends. The github.com/gocolly/redisstorage package provides a Redis-backed queue that multiple Colly instances can consume from:

import (
    "github.com/gocolly/colly/v2/queue"
    "github.com/gocolly/redisstorage"
)

redisStorage := &redisstorage.Storage{
    Address:  "redis:6379",
    Password: "",
    DB:       0,
    Prefix:   "colly_scraper",
}

q, _ := queue.New(3, redisStorage)
defer redisStorage.Client.Close()

q.AddURL("https://example.com")
q.Run(c)

With Redis storage, you can horizontally scale: launch 5 containers, each running a Colly instance pointed at the same Redis queue. URLs are distributed across workers, and deduplication is shared—no two workers scrape the same URL. This pattern scales to 100+ concurrent sessions with residential proxies, giving you aggregate throughput of 50+ requests per second across a fleet.

For containerization, the standard approach is a multi-stage Dockerfile with a scratch or alpine final image. Each container runs one Colly process, connects to a shared Redis instance, and uses ProxyHat's gateway for proxy rotation. See the ProxyHat documentation for gateway configuration details and pricing for residential proxy plans.

When Colly Isn't the Right Tool

Colly is an HTTP client scraper. It does not execute JavaScript. If your target is a single-page application (SPA) that renders content client-side via React, Vue, or Angular, Colly will see an empty <div id="root"> and nothing else. In these cases, you need a headless browser—Playwright, Puppeteer, or Chromium with a Go wrapper like chromedp.

You can still use ProxyHat proxies with browser-based tools. The proxy configuration goes on the browser launch flags (--proxy-server=http://gate.proxyhat.com:8080) or via a proxy authentication extension. The rotation strategy changes—browsers maintain connections longer, so sticky sessions become more important, and you rotate at the browser instance level rather than per-request.

For SERP tracking and static-content scraping, Colly with residential proxies is the right tool: fast, lightweight, and idiomatic Go.

Key Takeaways

• Use c.SetProxyFunc() with a custom func(*http.Request) (*url.URL, error) for dynamic, per-request proxy rotation—this is Colly's idiomatic extension point.
• Residential proxies are required for hard targets. Rotate -country- and -session- flags in the ProxyHat username for geo-targeting and stickiness.
• Always set Limit() rules with Parallelism, Delay, and RandomDelay to make traffic patterns look organic.
• Use OnError with r.Request.Retry() for transient failures (429, 5xx). Clone collectors for retry with modified settings.
• Scale horizontally with Redis-backed queues for distributed, multi-container scraping.
• Colly doesn't execute JavaScript. For SPAs, use a headless browser with proxy flags instead.