The Verge website cannot be crawled. Does anyone have a solution?

[socekin]

I can't crawl The Verge using crawl4ai. Does anyone have a solution?

[aravindkarnam]

@socekin Can you expand on this a little bit more. What do you mean by can't crawl. Can you share some screenshots, urls or code snippets, logs etc.

[socekin]

URL: https://www.theverge.com/news/622502/faa-orders-staff-find-funding-starlink-verizon-deal

The parameters：
{
"urls": "https://www.theverge.com/news/622502/faa-orders-staff-find-funding-starlink-verizon-deal",
"priority": 10,
"word_count_threshold": 100,
"exclude_external_links": "true",
"only_text": "true",
"magic": "true",
"markdown_generator": {
"content_filter": {
"type": "pruning", // "pruning" "bm25"
"threshold": 0.48,
"threshold_type": "dynamic",
"min_word_threshold": 20
}
}
}

This is the result of the crawl
response.json

The crawled result does not have main article content.