[Bug]: Unable to crawl https://www.behance.net

[Mng-dev-ai]

crawl4ai version

0.5.0.post4

Expected Behavior

it should return some results, tested the same url using firecrawl and it worked

Current Behavior

no results

Is this reproducible?

Yes

Inputs Causing the Bug

Steps to Reproduce

Code snippets

import asyncio
from crawl4ai import AsyncWebCrawler
from crawl4ai.async_configs import BrowserConfig, CrawlerRunConfig

async def main():
    browser_config = BrowserConfig()
    run_config = CrawlerRunConfig()

    async with AsyncWebCrawler(config=browser_config) as crawler:
        result = await crawler.arun(
            url="https://www.behance.net/",
            config=run_config
        )
        print(result.markdown)

if __name__ == "__main__":
    asyncio.run(main())

OS

macOS

Python version

3.10.2

Browser

No response

Browser version

No response

Error logs & Screenshots (if applicable)

[aravindkarnam]

@Mng-dev-ai I tried your code snippet as well on behance.net. Issue is that behance is loading all the data in the page via a javascript bundle and AJAX requests. Since you haven't explicitly added any wait condition, the scraping occurs soon as the DOM content is loaded(which is just a script tag). Try the following it should get you all the data from the page:

import asyncio
from crawl4ai import AsyncWebCrawler
from crawl4ai.async_configs import BrowserConfig, CrawlerRunConfig


async def main():
    browser_config = BrowserConfig(headless=False)
    run_config = CrawlerRunConfig(
        delay_before_return_html=3, scan_full_page=True, scroll_delay=1
    )

    async with AsyncWebCrawler(config=browser_config) as crawler:
        result = await crawler.arun(url="https://www.behance.net/", config=run_config)
        print(result.markdown)


if __name__ == "__main__":
    asyncio.run(main())

Compared to Firecrawl, we offer a lot more customisation and extensibility in the way pages are scraped, cleaned and data extracted. Our default crawler is optimised for static sites like wikipedia etc

So what I have done in the script above is

1. I'm asking the crawler to wait for 3 seconds, before it proceeds with scraping ( you could also customise this better by using wait_for instead, where you can use either javascript (an even happens) or css based conditions(a element with a specific id or class name either appears or disappears). For example, it's common to have a loader (like a spinning wheel), you can ask the crawler to proceed with scraping after loader disappears using wait_for.
2. Asking the crawler to scan full page, this is useful for pages that load data on scroll
3. Added 1 second scroll delay between consecutive scrolls.

This should work fine for most single page applications(SPA), however I see that we can improve user experience by toggling the run configs, based on nature of the page.

[Mng-dev-ai]

@aravindkarnam thank you, I think the problem was related to the browser, changed browser_type to firefox fixed the issue
but now I see another issue where extracted_content returns None

here's an example:

llm_strategy = LLMExtractionStrategy(
                llm_config=LLMConfig(
                    provider="gemini/gemini-2.0-flash-lite-preview-02-05",
                    api_token=os.getenv("GEMINI_API_KEY"),
                ),
                schema=PageContent.model_json_schema(),
                extraction_type="schema",
                instruction= instruction
                chunk_token_threshold=1000,
                overlap_rate=0.0,
                apply_chunking=True,
                input_format="markdown",
                extra_args={"temperature": 0.0, "max_tokens": 1000},
            )

            crawl_config = CrawlerRunConfig(
                extraction_strategy=llm_strategy,
                delay_before_return_html=3,
                scan_full_page=True,
                scroll_delay=1,
            )
            browser_config = BrowserConfig(browser_type="firefox",headless=True)
            async with AsyncWebCrawler(config=browser_config) as crawler:
                result = await crawler.arun(
                    url="https://www.behance.net",
                    config=crawl_config,
                )
                if result.success:
                    print(result.markdown) // this returns the data
                    print(result.extracted_content) // this returns None