Blocked by Cloudflare when crawling, but Firecrawl works

[dvirginz]

We’re seeing Cloudflare blocks when attempting to crawl the following URL using crawl4ai:

👉 https://www.idbny.com/

The response we get looks like this:

Cloudflare Ray ID: XXXX • Your IP: Click to reveal YYYY • Performance & security by Cloudflare

Interestingly, when using Firecrawl, we’re able to scrape the page without issues.

Anyone have ideas on how to handle this or what might be causing the discrepancy between the two tools? Curious if there’s something specific we could configure to bypass the Cloudflare block.