How to scrape from adidas page, how they detect its scraping

[cargt3]

Hi,

I'm building a RAG application and I need to scrape some pages for Markdown content. I'm having issues with the Adidas website. I’ve tried multiple paid web scraping solutions, but none of them worked. I also tried using Crawl4AI, and while it sometimes works, it's not reliable.

I'm trying to understand the actual bot detection mechanism used by the Adidas website. Even when I set headless=false and manually open the page using Chromium, I still get hit with an anti-bot challenge.

https://www.adidas.dk/hjaelp/returnering-refundering/returpolitik

regards

[onestardao]

hey, i’ve run into similar issues with adidas and a few other sites — they tend to use multiple layers of bot detection, not just headless flags. things like:

checking browser fingerprints (fonts, canvas, webgl, etc)

monitoring interaction patterns (timing, mouse/keyboard events)

ip reputation / rate limits

so even if you use chromium with headless=false, it can still trigger challenges if the fingerprint or behavior doesn’t match a “normal” browser session.

i’ve put together a simple problem map / checklist that covers the common detection angles (headers, tls, browser signals, etc). if you’d like, i can share the link — might save you some trial and error.