Disclaimer. This post documents an investigation of Akamai Bot Manager on the public site businesswire.com. The purpose is educational: to show how the defence is wired and how its layers look from the client side. All data sources used here (the MRSS feed and the S3 sitemap) are published by businesswire themselves in robots.txt for crawlers and search engines, and the work touches only publicly available content. If you apply the techniques described below to your own tasks, respect the site's robots.txt, its Terms of Service, and applicable law (CFAA in the US, GDPR and the Database Directive in the EU).
In our previous post we walked through what Akamai Bot Manager checks in 2026 across five parallel layers, and which stacks close each layer. This one is a concrete case for a real task: every fifteen minutes, pull new press releases from businesswire.com that match a query list, save the full HTML and metadata.
businesswire.com is a distribution platform for corporate press releases. The site sits behind Akamai. At first glance it looks like a reasonable target for scraping: there's a search modal in the header, listing pages, open URLs. In practice none of those three things behaves the way you'd expect.
The end result is a pipeline that combines Patchright and curl_cffi over a US sticky residential proxy, with two sources that businesswire themselves publish for crawlers: an MRSS feed and an S3 sitemap. One pass with five queries lands at 47 seconds, with no failed downloads. The full code is open at github.com/geekproxy/businesswire-scraper.
This post walks the path bottom-up: what doesn't work and why, then what works.
Endpoint map: what Akamai serves and what it blocks
Every request below goes through a residential US proxy with curl_cffi and chrome131 impersonation. That closes the TLS and HTTP/2 fingerprints (layer 1 of Akamai). A datacenter IP would be rejected on layer 4 (IP reputation) before TLS even starts.
GET / → 200, 446 KB ✓
GET /newsroom → 200, 715 b "Page Unavailable"
GET /newsroom/industry → 200, 442 KB ✓
GET /newsroom/subject → 200, 396 KB ✓
GET /newsroom/language → 200, 358 KB ✓
GET /news/home/{id}/en/... → 200, 2.4 KB Akamai sensor challenge
GET /sitemap.xml → 200, 46 KB ✓ (static pages only)
GET /sitemap-index.xml → 200, 1.5 KB ✓
GET /robots.txt → 200, 6 KB ✓Two non-obvious points stand out.
The root /newsroom (the page that hosts the search interface) returns a 715-byte stub titled "Page Unavailable" with an obfuscated Akamai JS challenge. At the same time /newsroom/industry, /subject and /language (sub-routes of the same page, used as filtered listings) are open and serve full 350–450 KB SPA bundles. This is typical Akamai endpoint-level policy: the root search route is locked down harder than the listings around it.
The release pages themselves (/news/home/{id}/...) return 2.4 KB. It's a challenge shell that sets a cookie and would normally redirect. Without a valid _abck (the Akamai cookie issued after a successful sensor_data POST) you do not get real HTML.
The /newsroom search, the most direct path for our task, is closed entirely. Only the listing sub-routes and, with a valid _abck, the release pages themselves are reachable.
What does not work
Plain requests with a spoofed User-Agent
import requests
r = requests.get("https://www.businesswire.com/",
headers={"User-Agent": "Mozilla/5.0 ..."})
# HTTP 403The TLS fingerprint of Python's urllib3 has nothing in common with Chrome. Akamai classifies it on layer 1 as "not a browser" before the server even reads HTTP headers. The spoofed User-Agent is irrelevant.
Vanilla Playwright (headless Chromium)
await page.goto("https://www.businesswire.com/")
# net::ERR_HTTP2_PROTOCOL_ERRORFails on the TLS and HTTP/2 handshake on the homepage itself. Headless Chromium has recognisable layer-2 markers (SETTINGS frame parameters, ALPN), and Akamai responds not with a 403 but by tearing down the stream at the h2 level, hence ERR_HTTP2_PROTOCOL_ERROR. None of --disable-features=Http2, --headless=new, or the playwright-stealth plugin fix this. The stealth plugin patches layer 3 (DOM properties, JS environment), and we never reach that layer.
--- default headless | args=[]
FAIL: net::ERR_HTTP2_PROTOCOL_ERROR
--- headless=new
FAIL: net::ERR_HTTP2_PROTOCOL_ERROR
--- disable-h2-features
FAIL: net::ERR_HTTP2_PROTOCOL_ERROR
--- disable-blink-auto
FAIL: net::ERR_HTTP2_PROTOCOL_ERROR
--- combo
FAIL: net::ERR_HTTP2_PROTOCOL_ERRORReproducible via probes/01_baseline_chromium.py.
Scraping the /newsroom search page directly
Even after a full warmup (homepage, mouse motion, a visit to one release page) /newsroom still returns the 715-byte stub. Akamai runs a stricter sensor on this route, and consecutive attempts from the same sticky IP get a 429. The root search route is simply protected harder than the listings around it.
The header search modal — it's not a real search
Every page on businesswire has a magnifier icon in the header. Click it, type "nasdaq" and a list of five recent releases pops up. It looks like a normal search, the kind you would expect to be reverse-engineerable.
Open the homepage in Patchright, hook context.on("response", ...) to capture every response, click the magnifier, type the query, wait 15 seconds after Enter. You get 92 captured network entries. Three of them look search-related:
POST 200 https://1dfqiezxuz-1.algolianet.com/1/indexes/*/queries
POST 200 https://1dfqiezxuz-2.algolianet.com/1/indexes/*/queries
POST 200 https://1dfqiezxuz-3.algolianet.com/1/indexes/*/queriesAlgolia. App ID 1DFQIEZXUZ, public search key 3ef401672a314ef0434873a57f64bfd3 sitting right in the homepage HTML. Request body is a multi-index batch:
{"requests": [
{"indexName": "prod_api::home.home", "query": "nasdaq"},
{"indexName": "prod_api::contacts.contacts", "query": "nasdaq"},
{"indexName": "prod_api::distribution.distribution", "query": "nasdaq"},
{"indexName": "prod_api::help.help", "query": "nasdaq"},
...
]}In the response every index returns an empty hits: []. Algolia found nothing. And yet the modal's DOM has 45 /news/home/{id}/... links with titles. Where do they come from?
A client-side filter. The Next.js page embeds a list of about ten recent releases in its props (used by the "Latest News" sidebar), and the search modal simply walks that list with a substring filter against whatever the user typed. There is no backend search for press releases at all. Algolia indexes only the content pages of the site (help, distribution, contacts).
So there is nothing to reverse-engineer. businesswire has no public search API for press releases. If your task is to find a specific release by substring, you either stand up an external search (Google News with site:businesswire.com, or a paid bypass service), or you look for a different source entirely.
Script: probes/04_reverse_search_modal.py.
The turning point: robots.txt
A standard move that everyone who has ever scraped a production site applies, and which somehow rarely shows up in bypass guides: open /robots.txt and read it before you start breaking the defence.
$ curl https://www.businesswire.com/robots.txt | grep -i sitemap
Sitemap: https://www.businesswire.com/sitemap-index.xml
Sitemap: https://bw-prod-sitemap.s3.us-east-1.amazonaws.com/webdmz1.vaprod.businesswire.com/home/smap-bw-mod.xml
Sitemap: https://bw-prod-sitemap.s3.us-east-1.amazonaws.com/webdmz1.vaprod.businesswire.com/gn-home/gn-bw-mod.xml
Sitemap: https://feed.businesswire.com/mrss/home/?rss=G1QFDERJXkJcFVJYWQ==The whole task is right there. businesswire themselves publish three alternative sources for Google News, crawlers and SEO indexers:
feed.businesswire.com/mrss/home/?rss=G1QFDERJXkJcFVJYWQ==— an MRSS feed on a separate subdomain. Returns the ~50 most recent releases with full title, description, link, and pubDate. This host is also behind Akamai, butcurl_cffi+chrome131over a residential proxy passes it cleaner than the main domain.bw-prod-sitemap.s3.us-east-1.amazonaws.com/.../smap-bw-mod.xml— a sitemap index on a public AWS S3 bucket. No Akamai here at all; any HTTP client reads it. The index links to daily sitemaps named2026-05-01.xml.gz(gzipped XML), each holding ~450 releases for that day.bw-prod-sitemap.s3.us-east-1.amazonaws.com/.../gn-home/gn-bw-mod.xml— same content in Google News format.
This is a public crawler interface. Nobody hides it. People just don't open robots.txt before drawing TLS-bypass diagrams.
The final pipeline
┌──────────────────────────────────────────────────┐
│ STEP 1: MRSS feed │
│ GET https://feed.businesswire.com/mrss/home/?rss=│
│ via rotating residential US + curl_cffi │
│ with impersonate=chrome131 │
│ → XML with ~50 latest releases │
│ (title, description, link, pubDate) │
└────────────┬─────────────────────────────────────┘
▼
┌──────────────────────────────────────────────────┐
│ STEP 2: substring filter over queries │
│ for item: │
│ if any(q.lower() in (title+description).lower()│
│ for q in queries): │
│ keep │
└────────────┬─────────────────────────────────────┘
▼
┌──────────────────────────────────────────────────┐
│ STEP 3: SQLite dedup → only new release_id's │
└────────────┬─────────────────────────────────────┘
▼
┌──────────────────────────────────────────────────┐
│ STEP 3.5: probe sticky port pool │
│ for each port → quick GET to businesswire.com │
│ pick the first one answering 200 + >50 KB <8s │
└────────────┬─────────────────────────────────────┘
▼
┌──────────────────────────────────────────────────┐
│ STEP 4: Patchright warmup on the chosen port │
│ homepage → mouse motion → one release │
│ collect _abck, ak_bmsc, bm_sv, bm_so │
└────────────┬─────────────────────────────────────┘
▼
┌──────────────────────────────────────────────────┐
│ STEP 5: curl_cffi(chrome131) + cookies + sticky │
│ parallel (5 threads) download of release pages. │
│ On ≥50% timeouts → repeat steps 3.5 and 4. │
└────────────┬─────────────────────────────────────┘
▼
┌──────────────────────────────────────────────────┐
│ STEP 6: HTML + JSON metadata to disk, │
│ insert into SQLite │
└──────────────────────────────────────────────────┘Once a day, or after a longer downtime, you can run scraper.py --backfill 2026-05-15. The script pulls the S3 daily sitemap for that date, filters URL slugs against queries, and pushes missing releases through the same warmup and curl_cffi pipeline.
Two-stage bypass for release pages
The MRSS feed under curl_cffi with chrome131 impersonation is just XML and opens cleanly. The release pages (/news/home/{id}/...) are in the first category of Akamai sites from our previous post: they need a real _abck, issued by a browser after a successful sensor_data POST.
The recipe is the same two-stage flow.
Stage 1: Patchright warmup. Patchright is a Playwright fork with binary-level Chromium stealth patches. Open the homepage through a sticky US residential session, simulate mouse motion and scroll (this feeds the behavioural layer), navigate to one release page, collect cookies.
async with async_playwright() as p:
browser = await p.chromium.launch(headless=True, channel="chromium",
args=["--no-sandbox"])
context = await browser.new_context(
proxy={"server": "http://rs.geekproxy.io:10000",
"username": "USER__cr.us", "password": "PASS"},
user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/131.0.0.0 Safari/537.36",
viewport={"width": 1366, "height": 900},
locale="en-US",
timezone_id="America/New_York",
)
page = await context.new_page()
await page.goto("https://www.businesswire.com/", wait_until="domcontentloaded")
await page.wait_for_timeout(12000) # let the sensor run
await page.mouse.move(120, 200)
await page.mouse.move(400, 350, steps=12)
await page.mouse.wheel(0, 600)
await page.wait_for_timeout(1500)
await page.goto(SAMPLE_RELEASE_URL, wait_until="domcontentloaded")
await page.wait_for_timeout(5000)
cookies = await context.cookies()From the collected cookies we care about _abck, ak_bmsc, bm_sv and bm_so. Akamai sets all four after a successful sensor_data POST.
Stage 2: curl_cffi bulk download. Create a Session(impersonate="chrome131"), copy the cookies in, and route through the same sticky proxy. The IP must not change inside one logical flow.
from curl_cffi import requests as ccffi
proxy = "http://USER__cr.us:[email protected]:10000"
s = ccffi.Session(impersonate="chrome131", timeout=30,
proxies={"http": proxy, "https": proxy})
for c in cookies:
if c["domain"].endswith("businesswire.com"):
s.cookies.set(c["name"], c["value"], domain=c["domain"])
r = s.get(release_url, headers={
"Accept-Language": "en-US,en;q=0.9",
"Referer": "https://www.businesswire.com/",
"Sec-Fetch-Site": "same-origin",
"Sec-Fetch-Mode": "navigate",
...
})
# HTTP 200, ~400 KB of real articleOne important point about matching browsers between the two stages. The curl_cffi library can imitate TLS fingerprints of different browsers: Chrome, Firefox, Safari, Edge, in different versions. If stage 1 uses a Chromium fork (Patchright), stage 2 in curl_cffi must imitate Chrome, for example chrome131. If stage 1 uses a Firefox-based fork like Camoufox, stage 2 must imitate Firefox (firefox-NNN). When Akamai issues _abck, it remembers which browser ran the sensor, and on subsequent requests it compares the announced TLS fingerprint. A Chrome cookie with a Firefox handshake on the next request is a clean mismatch signal, and Akamai blocks.
Five parallel threads through ThreadPoolExecutor is the right number for a single sticky IP. More than that and the behavioural layer starts treating the session as suspicious.
Reproducible: probes/03_two_stage_bypass.py.
Sticky session as a variable, not a constant
In theory, a residential sticky session means one IP for 5 to 120 minutes. In practice every residential provider occasionally hands you a problematic IP:
- a cold IP, a home router that has been offline for hours but is still listed in the pool;
- a tarpitted IP, where Akamai is deliberately slowing responses to that specific address because it was flagged earlier;
- a loaded link, where the residential connection is saturated.
When your sticky session lands on one of these, every request through it crawls or hangs for the rest of the session. A single timeout is noise. A chain of them is a signal.
On Geekproxy.io there are two independent ways to manage a pool of sticky sessions.
The first one is by port. Each port in the 10000–10010 range is a separate sticky session with its own IP. That gives eleven parallel slots: define the range in config, and the script walks them before warmup.
The second one is the sessid parameter in the login. Appending ;sessid.<any string or number> to the username pins the session to whichever IP that identifier currently represents, for about 30 minutes on average. Change the identifier and you get another parallel session: sessid.a1, sessid.a2, and so on. This is useful when you need more slots than ports available, or when you want to attach a known name to each session (handy for logging).
For our scraper I picked the port pool as the primary mechanism. It shows up cleanly in logs (you can see which port served which release), and eleven slots are enough for a 15-minute interval. The sessid option stays available — the config layout doesn't prevent using it if you ever run out of ports.
Before warmup the script walks the pool, makes a quick GET on businesswire.com on each port, and picks the first one that comes back with 200 + >50 KB in <8 seconds.
def pick_healthy_sticky_port(cfg, log):
for port in cfg.proxy_sticky_ports:
proxy = f"http://...:{port}"
t0 = time.time()
try:
r = ccffi.get(cfg.proxy_health_url,
impersonate="chrome131",
proxies={"http": proxy, "https": proxy},
timeout=cfg.proxy_health_timeout)
except Exception:
continue
if r.status_code == 200 and len(r.text) > 50000:
return port
return NoneFrom a real log of one pass:
sticky probe port 10000: Timeout (8.0s)
sticky probe port 10001: HTTP 200 447924 bytes (2.4s) — picked
warmup: launching Patchright on sticky port 10001
warmup: 32 cookies (Akamai: ['ak_bmsc', 'bm_s', 'bm_so', 'bm_sv'])
saved 2/2 ... pass elapsed 47.4sPort 10000 hung, the script picked 10001 in 2.4 seconds, the rest of the pass ran clean. Without the pool every download would have waited 30 seconds before timing out.
If more than half of in-flight downloads start timing out mid-pass, the script repeats the probe and warmup on a fresh session, then retries only the failed items.
Numbers
| Scenario | Time |
|---|---|
| One pass, MRSS + 5 queries, 0–2 new releases | 30–60 s |
| One pass, MRSS + 20 queries, 2–6 new releases | 45–110 s |
| Backfill for one day, ~450 sitemap URLs, 10–30 matches | 5–10 min |
| Warmup on its own | 30–35 s |
| Stage-2 download of one page via curl_cffi | 0.5–1.5 s |
On our test bench: 5/5 and 2/2 with zero failures. Cookies from a single warmup hold across dozens of sequential requests during one pass. Akamai does not flag.
When it breaks
A short forecast of what is likely to give first, and how to handle each case.
If Patchright stops producing _abck, that means Akamai has updated its sensor or started detecting Chromium stealth on a new parameter. The first thing to try is upgrading patchright. If that doesn't help, switch to Camoufox (a Firefox-based stealth build) and the paired impersonate="firefox-NNN" in curl_cffi. The rule about matching the browser engine on stage 1 with the TLS impersonation on stage 2 still applies.
If the MRSS feed URL changes, the live link is always in robots.txt on businesswire.com as a Sitemap: line pointing to feed.businesswire.com. The scraper reads it from source.mrss_url in the config, so it's a one-line update.
If feed.businesswire.com starts returning 403 or 429, point proxy.rotating_port at a port from the sticky pool so the fetch uses a stable IP. If that fails too, switch the primary source to the S3 sitemap. It has no title or description, only URLs, but the slug usually contains the headline and that's enough for substring matching.
If the sticky pool itself degrades (several sessions in a row return slow or empty responses), widen sticky_port_range to request more slots from the provider, or change proxy.country.
What's universal beyond businesswire
Most of what's in this post ports almost one to one to any Akamai-protected site with public content.
robots.txt is a first step, not a last resort. Sites often list a feed or sitemap on a separate subdomain or CDN that sits outside the main defence. The check costs five seconds and saves hours.
Map the protection per endpoint. Akamai's defence is not uniform: the root search route is usually stricter than the listings around it, and sitemaps or RSS are often open. Probe each endpoint individually with curl_cffi before you reach for Patchright.
Intercept network calls in the browser. Five minutes of watching XHR traffic tells you whether you're looking at a real backend search or a client-side filter. That check has saved many people weeks of reverse-engineering something that wasn't there.
Use the two-stage flow (Patchright for warmup, curl_cffi for bulk download) on any strict Akamai site with release-style content (news, articles, product catalogs).
Treat the sticky session as a pool rather than a single connection. Regardless of provider, residential addresses periodically misbehave. A plan B (next sticky slot, or a different sessid) belongs in the code from the very first commit.
Repository
github.com/geekproxy/businesswire-scraper — full code, configs, the four probe scripts that document the journey above. Quickstart:
git clone https://github.com/geekproxy/businesswire-scraper.git
cd businesswire-scraper
python3 -m venv venv && ./venv/bin/pip install -r requirements.txt
./venv/bin/patchright install chromium
cp config.yaml.example config.yaml
cp .env.example .env
# fill BW_PROXY_USERNAME, BW_PROXY_PASSWORD in .env
./venv/bin/python3 scraper.pyThe residential proxies behind this case are ours — geekproxy.io. Sticky sessions on separate ports, US/EU/AS geo pinning, ASNs from real ISPs. They drop into the pipeline above with no changes.