0افehknqtwz░◈Ξ∞xcلgjmpsvy▒◇∆π1بdفilorux▓◆∇∑0افehknqtwz░◈Ξ∞xcلgjmpsvy▒◇∆π1بdفilor

dفilorux▓◆∇∑0افehknqtwz░◈Ξ∞xcلgjmpsvy▒◇∆π1بdفilorux▓◆∇∑0افehknqtwz░◈Ξ∞xcلgjmpsvy

jmpsvy▒◇∆π1بdفilorux▓◆∇∑0افehknqtwz░◈Ξ∞xcلgjmpsvy▒◇∆π1بdفilorux▓◆∇∑0افehknqtwz░◈

qtwz░◈Ξ∞xcلgjmpsvy▒◇∆π1بdفilorux▓◆∇∑0افehknqtwz░◈Ξ∞xcلgjmpsvy▒◇∆π1بdفilorux▓◆∇∑0

x▓◆∇∑0افehknqtwz░◈Ξ∞xcلgjmpsvy▒◇∆π1بdفilorux▓◆∇∑0افehknqtwz░◈Ξ∞xcلgjmpsvy▒◇∆π1بd

◇∆π1بdفilorux▓◆∇∑0افehknqtwz░◈Ξ∞xcلgjmpsvy▒◇∆π1بdفilorux▓◆∇∑0افehknqtwz░◈Ξ∞xcلgj

∞xcلgjmpsvy▒◇∆π1بdفilorux▓◆∇∑0افehknqtwz░◈Ξ∞xcلgjmpsvy▒◇∆π1بdفilorux▓◆∇∑0افehknq

فehknqtwz░◈Ξ∞xcلgjmpsvy▒◇∆π1بdفilorux▓◆∇∑0افehknqtwz░◈Ξ∞xcلgjmpsvy▒◇∆π1بdفilorux

ilorux▓◆∇∑0افehknqtwz░◈Ξ∞xcلgjmpsvy▒◇∆π1بdفilorux▓◆∇∑0افehknqtwz░◈Ξ∞xcلgjmpsvy▒◇

psvy▒◇∆π1بdفilorux▓◆∇∑0افehknqtwz░◈Ξ∞xcلgjmpsvy▒◇∆π1بdفilorux▓◆∇∑0افehknqtwz░◈Ξ∞

wz░◈Ξ∞xcلgjmpsvy▒◇∆π1بdفilorux▓◆∇∑0افehknqtwz░◈Ξ∞xcلgjmpsvy▒◇∆π1بdفilorux▓◆∇∑0اف

◆∇∑0افehknqtwz░◈Ξ∞xcلgjmpsvy▒◇∆π1بdفilorux▓◆∇∑0افehknqtwz░◈Ξ∞xcلgjmpsvy▒◇∆π1بdفi

π1بdفilorux▓◆∇∑0افehknqtwz░◈Ξ∞xcلgjmpsvy▒◇∆π1بdفilorux▓◆∇∑0افehknqtwz░◈Ξ∞xcلgjmp

cلgjmpsvy▒◇∆π1بdفilorux▓◆∇∑0افehknqtwz░◈Ξ∞xcلgjmpsvy▒◇∆π1بdفilorux▓◆∇∑0افehknqtw

hknqtwz░◈Ξ∞xcلgjmpsvy▒◇∆π1بdفilorux▓◆∇∑0افehknqtwz░◈Ξ∞xcلgjmpsvy▒◇∆π1بdفilorux▓◆

orux▓◆∇∑0افehknqtwz░◈Ξ∞xcلgjmpsvy▒◇∆π1بdفilorux▓◆∇∑0افehknqtwz░◈Ξ∞xcلgjmpsvy▒◇∆π

vy▒◇∆π1بdفilorux▓◆∇∑0افehknqtwz░◈Ξ∞xcلgjmpsvy▒◇∆π1بdفilorux▓◆∇∑0افehknqtwz░◈Ξ∞xc

░◈Ξ∞xcلgjmpsvy▒◇∆π1بdفilorux▓◆∇∑0افehknqtwz░◈Ξ∞xcلgjmpsvy▒◇∆π1بdفilorux▓◆∇∑0افeh

∑0افehknqtwz░◈Ξ∞xcلgjmpsvy▒◇∆π1بdفilorux▓◆∇∑0افehknqtwz░◈Ξ∞xcلgjmpsvy▒◇∆π1بdفilo

بdفilorux▓◆∇∑0افehknqtwz░◈Ξ∞xcلgjmpsvy▒◇∆π1بdفilorux▓◆∇∑0افehknqtwz░◈Ξ∞xcلgjmpsv

gjmpsvy▒◇∆π1بdفilorux▓◆∇∑0افehknqtwz░◈Ξ∞xcلgjmpsvy▒◇∆π1بdفilorux▓◆∇∑0افehknqtwz░

nqtwz░◈Ξ∞xcلgjmpsvy▒◇∆π1بdفilorux▓◆∇∑0افehknqtwz░◈Ξ∞xcلgjmpsvy▒◇∆π1بdفilorux▓◆∇∑

ux▓◆∇∑0افehknqtwz░◈Ξ∞xcلgjmpsvy▒◇∆π1بdفilorux▓◆∇∑0افehknqtwz░◈Ξ∞xcلgjmpsvy▒◇∆π1ب

▒◇∆π1بdفilorux▓◆∇∑0افehknqtwz░◈Ξ∞xcلgjmpsvy▒◇∆π1بdفilorux▓◆∇∑0افehknqtwz░◈Ξ∞xcلg

rehan@neural-mesh :~

$ ▊

rehan.osRehan Rao

Data · Scraping 2025 · 06 7 min

Proxy rotation algorithms for hostile web scraping.

Sticky sessions, IP fingerprints, adaptive backoff. A field-tested rotation strategy that scrapes millions of pages without triggering hostile counter-measures.

Rehan Rao

AI & Backend Systems Engineer

architecturerotation.loop

Modern anti-bot stacks are stateful. They correlate IP, TLS fingerprint, timing, and behavior. Beating them at scale is not about buying more proxies — it is about scheduling them coherently.

The threat model

IP reputation scoring (residential > mobile > datacenter).
TLS + HTTP/2 fingerprint clustering (JA3, akamai_fp).
Behavioral timing signatures — humans are not Poisson.
Session cookies bound to first-seen IP class.

Pool topology

Split proxies into pools by class. Never mix a residential and a datacenter IP inside the same session — that mismatch alone is a fingerprint. Each pool exposes the same interface: acquire(), release(ok=True|False).

The scheduler

Weighted round-robin with health scoring. Every proxy carries an EWMA of success rate over the last N requests; the scheduler biases toward the top quartile and quarantines anything under 60%.

def pick(pool):
    healthy = [p for p in pool if p.ewma > 0.6]
    weights = [p.ewma ** 2 for p in healthy]
    return random.choices(healthy, weights=weights, k=1)[0]

Fingerprints

Pair every proxy with a stable fingerprint bundle: user-agent, TLS profile (via curl_cffi), viewport, timezone. Rotate the bundle only when you rotate the IP class — never mid-session.

Adaptive backoff

On 429 or 403, quarantine the proxy for exponential time and inject jitter into the global request cadence. Do not retry immediately with a different proxy — that pattern itself is detectable.

note

The single biggest win is humanizing cadence: log-normal think times between requests, not uniform sleep. Anti-bot heuristics look for evenly spaced traffic first.