Cookies Popups & Few errors/bugs?

[an0mal1a]

Hey, I don't know if I'm the only one encountering these issues or if I'm doing something wrong with my implementation. I'm trying to handle a cookies popup before processing the page on a website like "Fotocasa.com". The goal is to scroll to the bottom to dynamically render all the content, but I'm facing several issues:

1. VirtualScrollConfig and scan_full_page dont wait for the page to load or respect the "wait_for parameter"

   • Even when I set wait_for to ensure the page is ready, the scroll starts prematurely, causing failures when dynamic content isn't loaded yet.

2. The js_code parameter doesn't seem to work in my context.

   • I'm trying to use js_code to click on the cookies popup button (id=didomi-notice-agree-button) and remove it from the DOM, but it doesn't execute as expected.

3. remove_overlay_elements doesn't remove the cookies popup effectively.

   • While this parameter works when taking screenshots, it doesn't seem to trigger before scrolling or extracting content, leaving the popup in place and blocking the scroll.

4. Errors when using wait_for with JavaScript conditions.

   • If the JavaScript condition in wait_for evaluates to null or undefined, it throws an error instead of handling the absence gracefully.

Additional Notes:

• I experimented with the wait_for_images parameter, and while it seemed to make the scroll work once, I couldn't replicate the behavior consistently.
  . My main goal is to scroll down the entire page to load all dynamic content.

I'm using Crawl4AI for proxy management and scalability in scraping.

Reproduce Errors:

• JavaScript Code on wait_for or js_code

If i use a condition like this:

wait_for="js:() => !document.querySelector('.cookie-popup-selector')"

The program raise an error trying to read properties of null/undefined, so i have to do this instead;

wait_for="js:() => !!document.querySelector('.cookie-popup-selector')"

Adding !! ensure the condition return true or false, but im unsure if this is the correct approach for the wait_for parameter.

Code

import json
import asyncio
from dotenv import load_dotenv
from pydantic import BaseModel, Field
from crawl4ai import (
    AsyncWebCrawler,
    BrowserConfig,
    CrawlerRunConfig,
    CacheMode,
    RoundRobinProxyStrategy,
    ProxyConfig,
    JsonCssExtractionStrategy,
)
load_dotenv()

# Scroll not waiting...

async def collect_all_pages(links_data):
    proxies = ProxyConfig.from_env()
    if not proxies: 
        raise ValueError("No proxies has been detected...")
    
    schema = {
        "name": "Pages",
        "baseSelector": "div.re-Pagination",
        "fields": [
            {
                "name": "page_nums",
                "selector": "li.sui-MoleculePagination-item",
                "type": "text",
                "subfields": [
                    {
                        "name": "page_num",
                        "selector": "li.sui-MoleculePagination-item",
                        "type": "text"
                    }
                ]
            }
        ]
    }

    extraction_strategy = JsonCssExtractionStrategy(schema)
    proxy_strategy = RoundRobinProxyStrategy(proxies)
    browser_config = BrowserConfig(headless=False, verbose=False)

    run_config = CrawlerRunConfig(
        # wait_for="div.re-Pagination",
        cache_mode=CacheMode.BYPASS,
        extraction_strategy=extraction_strategy,
        scan_full_page=True,
        remove_overlay_elements=True,
        js_code=[
            "document.querySelector('#didomi-notice-agree-button')?.click();",
        ],
        delay_before_return_html=2.5
    )

    for data in links_data.values(): 
        urls = data["links"]

        async with AsyncWebCrawler(config=browser_config) as crawler:
            print("Starting batch crawl with proxy rotation")
            result = await crawler.arun(
                url=urls[0],
                config=run_config
            )
            if result.success:
                print(result.extracted_content)

            else: 
                print("[ERROR] ...")


    return proxy_strategy

async def main():
    schema = {
        "name": "Properties Links",
        "baseSelector": "article",
        "fields": [
            {
                "name": "url",
                "selector": "a:first-of-type",
                "type": "attribute",
                "attribute": "href"
            },
        ]
    }

    extraction_strategy = JsonCssExtractionStrategy(schema)    
    browser_config = BrowserConfig(headless=True, verbose=False)

    # virtual_scroll_config = VirtualScrollConfig(
    #     container_selector="main",
    #     scroll_count=50,
    #     wait_after_scroll=5,
    #     scroll_by="container_height"
    # )
    
    # Setup Crawler Config
    run_config = CrawlerRunConfig(
        cache_mode=CacheMode.BYPASS,
        # proxy_rotation_strategy=proxy_strategy,
        extraction_strategy=extraction_strategy,
        wait_for="article",
        wait_for_images=True,
        scan_full_page=True,
        scroll_delay=0.3
    )

    links_data = {
        "sell_link": {
            "storage": [],
            "links": [
                  "https://www.fotocasa.es/es/comprar/viviendas/illes-balears-provincia/ibiza-eivissa/l/1",
                ]
        },

        "rent_link": {
            "storage": [],
            "links": [
                  "https://www.fotocasa.es/es/alquiler/viviendas/illes-balears-provincia/ibiza-eivissa/l/1",
                ]
        }
    }

    await collect_all_pages(links_data)
    exit() # testing this function

    for data in links_data.values():
        urlStorage = data["storage"]
        urls = data["links"]

        async with AsyncWebCrawler(config=browser_config) as crawler:
            print("Starting batch crawl with proxy rotation")
            results = await crawler.arun_many(
                urls=urls,
                config=run_config
            )
            for result in results:
                if result.success:
                    for link in json.loads(result.extracted_content):
                        link = link.get("url", "") 
                        if link:
                            urlStorage.append(link)      
                    
                else:
                    print(f"Error in {result.url}")

    print(json.dumps(links, indent=4))


asyncio.run(main())