Advanced Multi-URL Crawling with authentication first : is it possible ?

[ochaoui]

Hi folks,

I am writing a script that scraps a website. But before doing it, I have to authenticate, get the urls I want then do a multi-crawling.

    async def scrap_toto_to_markdown(
        self,
        urls: list[str],
    ) -> list[str]:
        """Scrape toto to markdown from the Little Big Connexion platform.

        Args:
            urls: List of TOTO URLs to scrape.

        Returns:
            A list of markdown content for each TOTO.
        """
        toto_markdown: list[str] = []

        browser_config = BrowserConfig(headless=HEADLESS, verbose=False)
        run_config = CrawlerRunConfig(
            cache_mode=CacheMode.BYPASS,
            stream=False,  # Default: get all results at once
            session_id="session1",
        )

        dispatcher = MemoryAdaptiveDispatcher(
            memory_threshold_percent=70.0,
            check_interval=1.0,
            max_session_permit=10,
            monitor=CrawlerMonitor(
                urls_total=len(urls),
                enable_ui=False,
            ),
        )

        async with AsyncWebCrawler(config=browser_config) as crawler:

            # Hook for authentication and route configuration
            async def on_page_context_created(
                page: Page, context: BrowserContext, **kwargs: dict[str, Any]
            ) -> Page:
                return await _login_on_page_context_created(
                    page, context, self.username, self.password, self.auth_link
                )

            # Set up hooks on the crawler strategy with class attributes
            crawler.crawler_strategy.set_hook(
                "on_page_context_created",
                on_page_context_created,
            )
            # Get all results at once
            results = await crawler.arun_many(
                urls=urls, config=run_config, dispatcher=dispatcher
            )

            # Process all results after completion
            for result in results:
                if result.success:
                    toto_markdown.append(result.markdown)
                else:
                    print(f"Failed to crawl {result.url}: {result.error_message}")

        return toto_markdown

As you can see, I added the hook on_page_context_created that handles the connection (it works I tested it in a single url scrap).

The problem with this implementation is that crawl4ai tries to authenticate for all the calls even if apparently they share all the same context.

For example I sent 3 urls, the first worked because it did the authentication. But for the other, it tries also to do the authentication but it fails because the goto page goes directly to the requested page.

Is there a way for the Multi-URL to make the authentication once for all the calls ?

Thanks in advance !