How to fix the 407 errors while scrapping the websites using crawl4ai + playwright + apify with proxies

[prokhorenkomykhailo]

Hello, community memebers, hope you are doing great.
Recently I have worked on Crawl4ai, playwright using proxy with Apify for crawling some web pages.

The main goal is to get html content and the screenshot of the webpages and their status code with desktop + mobile versions.
Here is the source code

from apify import Actor
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
import asyncio
import subprocess
import sys
import base64
from collections import defaultdict
 
# Install required system dependencies for Playwright
subprocess.check_call([sys.executable, "-m", "playwright", "install-deps"])
subprocess.check_call([sys.executable, "-m", "playwright", "install", "chromium"])
 
# Constants
BASE_URL = f"https://api.apify.com/v2/key-value-stores/{DATASET_ID}/records/"
 
# User agents for desktop and mobile
USER_AGENTS = {
    "desktop": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36",
    "mobile": "Mozilla/5.0 (iPhone; CPU iPhone OS 15_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.0 Mobile/15E148 Safari/604.1"
}               
 
# List of available proxy providers
PROXY_PROVIDERS = ["smartproxy", "goproxies", "brightdata"]
 
def get_proxy_config(country_code, proxy_provider="smartproxy"):
    """Retrieve proxy configuration for the specified country and proxy provider."""
    PROXY_CONFIGS = {
        "goproxies": {
            "proxy_host": "proxy.goproxies.com",
            "proxy_port": 1080,
            "proxy_username": f"customer-G_USER-country-{country_code.lower()}",
            "proxy_password": "XXX-YYY-ZZZ"
        },
        "brightdata": {
            "proxy_host": "brd.superproxy.io",
            "proxy_port": 33335,
            "proxy_username": f"brd-customer-BRIGHT_USER-zone-residential-country-{country_code.lower()}",
            "proxy_password": "XXX-YYY-ZZZ"
        },
        "smartproxy": {
            "proxy_host": f"{country_code.lower()}.smartproxy.com",
            "proxy_port": 24001,
            "proxy_username": "SMART_USERNAME",
            "proxy_password": "XXX-YYY-ZZZ"
        }
    }
 
    try:
        return PROXY_CONFIGS[proxy_provider]
    except KeyError:
        print(f"Warning: Invalid proxy_provider '{proxy_provider}'. Using default proxy configuration (smartproxy).")
        return PROXY_CONFIGS["smartproxy"]
 
async def save_datastore(data, dataset, proxy_type, country_code):
    """Save crawled data to the dataset."""
    try:
        base_key = f"{data['url'].replace('https://', '').replace('/', '_')}_{proxy_type}_{country_code}"
        print(f"Generated base key: {base_key}")
 
        # Save screenshots
        screenshot_key_desktop = f"{base_key}_desktop.jpg"
        if data['screenshot_url_desktop']:
            imagebytes = base64.b64decode(data['screenshot_url_desktop'])
            await dataset.set_value(screenshot_key_desktop, imagebytes, content_type='image/jpeg')
 
        screenshot_key_mobile = f"{base_key}_mobile.jpg"
        if data['screenshot_url_mobile']:
            imagebytes = base64.b64decode(data['screenshot_url_mobile'])
            await dataset.set_value(screenshot_key_mobile, imagebytes, content_type='image/jpeg')
 
        # Update URLs for saved files
        data['screenshot_url_desktop'] = BASE_URL + screenshot_key_desktop
        data['screenshot_url_mobile'] = BASE_URL + screenshot_key_mobile
        data['html_source_code_mobile'] = BASE_URL + f"{base_key}_mobile.html"
 
        # Save combined data
        total_obj_key = f"json_{base_key}_object"
        await dataset.set_value(total_obj_key, data)
        return data
 
    except Exception as e:
        print(f"Error saving data: {e}")
        return None
 
async def crawl_with_proxies(url, country_code, proxy_credentials, use_screenshot, device, agent, dataset, proxy_type):
    """Crawl a URL using the specified proxy and device configuration."""
    print(f"Starting crawl for {device} version of {url} in {country_code} using {proxy_type} proxy")
    try:
        proxy_config = {
            "server": f"http://{proxy_credentials['proxy_host']}:{proxy_credentials['proxy_port']}",
            "username": proxy_credentials["proxy_username"],
            "password": proxy_credentials["proxy_password"],
        }
 
        browser_config = BrowserConfig(
            browser_type="chromium",
            headless=True,
            viewport_width=375 if device == 'mobile' else 1280,
            viewport_height=812 if device == 'mobile' else 720,
            user_agent=agent,
            proxy_config=proxy_config,
            headers={
                "Accept-Language": "en-US,en;q=0.9",
                "Sec-CH-UA": '"Chromium";v="122", "Not(A:Brand";v="24", "Google Chrome";v="122"',
                "Sec-Fetch-Dest": "document",
                "Sec-Fetch-Mode": "navigate",
            },
            ignore_https_errors=True,
            extra_args=[  # ✅ Correct parameter name
                "--disable-web-security",
                "--disable-blink-features=AutomationControlled",
            ],
            text_mode=False,
            light_mode=False,
        )
 
 
        run_config = CrawlerRunConfig(
            cache_mode=CacheMode.BYPASS,
            screenshot=use_screenshot,
            wait_until='networkidle',  # Wait for network idle state
            wait_for_images=True,
            scan_full_page=True,
            screenshot_wait_for=30,  # Increased to 90 seconds for heavy pages
            scroll_delay=3,  # Increased delay between scrolls
            verbose=True,
            page_timeout=300000,  # 2 minutes timeout
            delay_before_return_html=5,
            log_console=True
            # max_scroll_height=25000,  # For pages with infinite scroll
        )
 
        async with AsyncWebCrawler(config=browser_config) as crawler:
            result = await crawler.arun(url, config=run_config)
            if not result.success:
                print(f"Failed to crawl {url}: {result.error_message}")
                return None
 
            return {
                'url': url,
                'device': device,
                'html_source_code': result.html,
                'http_status_code': result.status_code,
                'desktop_screenshot_url': result.screenshot,
                'final_url': result.redirected_url
            }
 
    except Exception as e:
        print(f"Error crawling {url}: {str(e)}")
        return None
 
async def attempt_crawl_with_proxy(country_code, base_url, dataset, proxy_provider, proxy_config):
    """Attempt to crawl with a specific proxy configuration."""
    url = base_url
    print(f"Attempting crawl with {proxy_provider} proxy for {url}")
 
    tasks = []
    for device, ua in USER_AGENTS.items():
        tasks.append(
            crawl_with_proxies(
                url=url,
                country_code=country_code,
                proxy_credentials=proxy_config,
                use_screenshot=True,
                device=device,
                agent=ua,
                dataset=dataset,
                proxy_type=proxy_provider
            )
        )
 
    responses = await asyncio.gather(*tasks)
    valid_responses = [response for response in responses if response is not None]
 
    if not valid_responses:
        return None
 
    grouped_data = defaultdict(lambda: {
        'url': None,
        'html_source_code_desktop': None,
        'html_source_code_mobile': None,
        'http_status_code_desktop': None,
        'http_status_code_mobile': None,
        'screenshot_url_desktop': None,
        'screenshot_url_mobile': None,
        'final_url': None,
    })
 
    for entry in valid_responses:
        url = entry['url']
        device = entry['device']
        
        if grouped_data[url]['url'] is None:
            grouped_data[url]['url'] = url
        
        if device == 'desktop':
            grouped_data[url]['html_source_code_desktop'] = entry['html_source_code']
            grouped_data[url]['http_status_code_desktop'] = entry['http_status_code']
            grouped_data[url]['screenshot_url_desktop'] = entry['desktop_screenshot_url']
            grouped_data[url]['final_url'] = entry['final_url']
        elif device == 'mobile':
            grouped_data[url]['html_source_code_mobile'] = entry['html_source_code']
            grouped_data[url]['http_status_code_mobile'] = entry['http_status_code']
            grouped_data[url]['screenshot_url_mobile'] = entry['desktop_screenshot_url']
            grouped_data[url]['final_url'] = entry['final_url']
 
    for response in grouped_data.values():
        if response:
            try:
                formatted_response = await save_datastore(response, dataset, proxy_provider, country_code)
                if formatted_response:
                    return_dict = {
                        'proxy': proxy_provider,
                        'proxy_country': country_code,
                        'screenshot': formatted_response['screenshot_url_desktop'],
                        'mobile_screenshot': formatted_response['screenshot_url_mobile'],
                        'status_code': formatted_response['http_status_code_desktop'],
                        'requested_url': url,
                        'final_url': formatted_response['final_url'],
                        'html': formatted_response['html_source_code_desktop'],
                    }
 
                    output_key = f"output_{country_code}_{url.replace('https://', '').replace('/', '_')}"
                    await Actor.set_value(key=output_key, value=return_dict)
                    await Actor.push_data([{'response': return_dict}])
 
                    status_message = f"SUCCESS - HTTP {formatted_response['http_status_code_desktop']} - {url} - TR scrape"
                    await Actor.set_status_message(status_message)
                    return return_dict
            except Exception as e:
                print(f"Error processing response: {e}")
                return None
    return None
 
async def process_country_with_retry(country_code, base_url, dataset, proxy_provider=None):
    """Process the URL for a specific country with retry logic."""
    print(f"Processing country: {country_code}")
 
    # Determine providers to try (specified first, then others)
    providers_to_try = []
    if proxy_provider:
        providers_to_try.append(proxy_provider)
    for provider in PROXY_PROVIDERS:
        if provider not in providers_to_try:
            providers_to_try.append(provider)
 
    # Try each provider with up to 3 retries
    for provider in providers_to_try:
        for attempt in range(3):
            print(f"Attempt {attempt+1}/3 with {provider} for {country_code}")
            proxy_config = get_proxy_config(country_code, provider)
            if not proxy_config:
                print(f"Invalid proxy config for {provider}, skipping...")
                break
 
            result = await attempt_crawl_with_proxy(
                country_code, base_url, dataset, provider, proxy_config
            )
            if result:
                return result
            print(f"Attempt {attempt+1} failed for {provider}")
 
    print(f"All attempts failed for {country_code}")
    return None
 
async def main():
    """Main function to handle crawling for a single country and URL."""
    async with Actor:
        dataset = await Actor.open_key_value_store(id=DATASET_ID)
        input_data = await Actor.get_input() or {}
 
        url = input_data.get("url")
        country = input_data.get("country")
        proxy_provider = input_data.get("proxy_provider")
 
        if not url or not country:
            error_message = "Missing required parameters 'url' or 'country'"
            await Actor.exit(status_message=error_message, exit_code=1)
 
        result = await process_country_with_retry(country, url, dataset, proxy_provider)
        if result:
            status_message = f"SUCCESS - HTTP {result['status_code']} - {url} - {country} scrape"
            await Actor.exit(status_message=status_message, exit_code=0)
        else:
            status_message = f"FAIL - No valid response for {url} - {country} scrape"
            await Actor.exit(status_message=status_message, exit_code=1)
 
asyncio.run(main())

It's working generally fine.
But it contains several issues - like

1. Cloudflare pages often appearing

2. Error (HTTP 407)
   

3. NO HTML, NO Screenshot Results from Crawl
   
   
   

This is the log in Apify

2025-03-18T16:41:19.480Z ACTOR: Pulling Docker image of build r1vuUg5TGLtJGSXlM from repository.
2025-03-18T16:41:25.945Z ACTOR: Creating Docker container.
2025-03-18T16:41:26.588Z ACTOR: Starting Docker container.
2025-03-18T16:41:26.828Z Starting X virtual framebuffer using: Xvfb :99 -ac -screen 0 1920x1080x24+32 -nolisten tcp
2025-03-18T16:41:26.830Z Executing main command
2025-03-18T16:41:28.150Z Downloading model definition files...
2025-03-18T16:41:28.389Z header-network.zip            OK!
2025-03-18T16:41:28.390Z input-network.zip             OK!
2025-03-18T16:41:28.391Z headers-order.json            OK!
2025-03-18T16:41:28.392Z browser-helper-file.json      OK!
2025-03-18T16:41:28.393Z fingerprint-network.zip       OK!
2025-03-18T16:41:29.985Z Installing dependencies...
2025-03-18T16:41:30.062Z 
0% [Working]
            
Get:1 http://deb.debian.org/debian bookworm InRelease [151 kB]
2025-03-18T16:41:30.073Z 
0% [1 InRelease 15.7 kB/151 kB 10%]
                                   
0% [Waiting for headers]
                        
Get:2 http://deb.debian.org/debian bookworm-updates InRelease [55.4 kB]
2025-03-18T16:41:30.074Z 
                        
Get:3 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
2025-03-18T16:41:30.143Z 
                        
0% [Working]
0% [Working]
            
Get:4 http://deb.debian.org/debian bookworm/main amd64 Packages [8792 kB]
2025-03-18T16:41:30.189Z 
0% [4 Packages 5122 B/8792 kB 0%]
                                 
0% [Working]
0% [4 Packages store 0 B]
0% [4 Packages store 0 B]
                         
Get:5 http://deb.debian.org/debian bookworm-updates/main amd64 Packages [13.5 kB]
2025-03-18T16:41:30.232Z 
0% [4 Packages store 0 B] [5 Packages 13.5 kB/13.5 kB 100%]
                                                           
0% [4 Packages store 0 B]
91% [4 Packages store 0 B]
                          
Get:6 http://deb.debian.org/debian-security bookworm-security/main amd64 Packages [249 kB]
2025-03-18T16:41:31.076Z 
91% [4 Packages store 0 B] [6 Packages 0 B/249 kB 0%]
                                                     
93% [4 Packages store 0 B]
93% [4 Packages store 0 B]
                          
96% [Working]
96% [5 Packages store 0 B]
                          
98% [Working]
98% [6 Packages store 0 B]
                          
100% [Working]
              
Fetched 9310 kB in 1s (8947 kB/s)
2025-03-18T16:41:31.632Z 
Reading package lists... 0%

Reading package lists... 0%

Reading package lists... 0%

Reading package lists... 95%

Reading package lists... 95%

Reading package lists... 95%

Reading package lists... 95%

Reading package lists... 98%

Reading package lists... 98%

Reading package lists... Done
2025-03-18T16:41:32.136Z 
Reading package lists... 0%

Reading package lists... 0%

Reading package lists... 0%

Reading package lists... 95%

Reading package lists... 95%

Reading package lists... 95%

Reading package lists... 95%

Reading package lists... 98%

Reading package lists... 98%

Reading package lists... Done
2025-03-18T16:41:32.281Z 
Building dependency tree... 0%

Building dependency tree... 0%

Building dependency tree... 50%

Building dependency tree... 50%

Building dependency tree... Done
2025-03-18T16:41:32.282Z 
Reading state information... 0% 

Reading state information... 0%

Reading state information... Done
2025-03-18T16:41:32.285Z libasound2 is already the newest version (1.2.8-1+b1).
2025-03-18T16:41:32.286Z libatk-bridge2.0-0 is already the newest version (2.46.0-5).
2025-03-18T16:41:32.287Z libatk1.0-0 is already the newest version (2.46.0-5).
2025-03-18T16:41:32.290Z libatspi2.0-0 is already the newest version (2.46.0-5).
2025-03-18T16:41:32.291Z libcairo2 is already the newest version (1.16.0-7).
2025-03-18T16:41:32.292Z libcups2 is already the newest version (2.4.2-3+deb12u8).
2025-03-18T16:41:32.292Z libdbus-1-3 is already the newest version (1.14.10-1~deb12u1).
2025-03-18T16:41:32.293Z libdrm2 is already the newest version (2.4.114-1+b1).
2025-03-18T16:41:32.294Z libgbm1 is already the newest version (22.3.6-1+deb12u1).
2025-03-18T16:41:32.295Z libglib2.0-0 is already the newest version (2.74.6-2+deb12u5).
2025-03-18T16:41:32.296Z libnspr4 is already the newest version (2:4.35-1).
2025-03-18T16:41:32.297Z libnss3 is already the newest version (2:3.87.1-1+deb12u1).
2025-03-18T16:41:32.298Z libpango-1.0-0 is already the newest version (1.50.12+ds-1).
2025-03-18T16:41:32.298Z libx11-6 is already the newest version (2:1.8.4-2+deb12u2).
2025-03-18T16:41:32.299Z libxcb1 is already the newest version (1.15-1).
2025-03-18T16:41:32.300Z libxcomposite1 is already the newest version (1:0.4.5-1).
2025-03-18T16:41:32.301Z libxdamage1 is already the newest version (1:1.1.6-1).
2025-03-18T16:41:32.302Z libxext6 is already the newest version (2:1.3.4-1+b1).
2025-03-18T16:41:32.303Z libxfixes3 is already the newest version (1:6.0.0-2).
2025-03-18T16:41:32.304Z libxkbcommon0 is already the newest version (1.5.0-1).
2025-03-18T16:41:32.305Z libxrandr2 is already the newest version (2:1.5.2-2+b1).
2025-03-18T16:41:32.305Z libcairo-gobject2 is already the newest version (1.16.0-7).
2025-03-18T16:41:32.306Z libdbus-glib-1-2 is already the newest version (0.112-3).
2025-03-18T16:41:32.308Z libfontconfig1 is already the newest version (2.14.1-4).
2025-03-18T16:41:32.308Z libgdk-pixbuf-2.0-0 is already the newest version (2.42.10+dfsg-1+deb12u1).
2025-03-18T16:41:32.309Z libgtk-3-0 is already the newest version (3.24.38-2~deb12u3).
2025-03-18T16:41:32.310Z libharfbuzz0b is already the newest version (6.0.0+dfsg-3).
2025-03-18T16:41:32.311Z libpangocairo-1.0-0 is already the newest version (1.50.12+ds-1).
2025-03-18T16:41:32.312Z libx11-xcb1 is already the newest version (2:1.8.4-2+deb12u2).
2025-03-18T16:41:32.313Z libxcb-shm0 is already the newest version (1.15-1).
2025-03-18T16:41:32.313Z libxcursor1 is already the newest version (1:1.2.1-1).
2025-03-18T16:41:32.314Z libxi6 is already the newest version (2:1.8-1+b1).
2025-03-18T16:41:32.315Z libxrender1 is already the newest version (1:0.9.10-1.1).
2025-03-18T16:41:32.316Z libxtst6 is already the newest version (2:1.2.3-1.1).
2025-03-18T16:41:32.317Z libsoup-3.0-0 is already the newest version (3.2.2-2).
2025-03-18T16:41:32.318Z gstreamer1.0-libav is already the newest version (1.22.0-2).
2025-03-18T16:41:32.319Z gstreamer1.0-plugins-bad is already the newest version (1.22.0-4+deb12u5).
2025-03-18T16:41:32.319Z gstreamer1.0-plugins-base is already the newest version (1.22.0-3+deb12u4).
2025-03-18T16:41:32.320Z gstreamer1.0-plugins-good is already the newest version (1.22.0-5+deb12u2).
2025-03-18T16:41:32.321Z libegl1 is already the newest version (1.6.0-1).
2025-03-18T16:41:32.322Z libenchant-2-2 is already the newest version (2.3.3-2).
2025-03-18T16:41:32.323Z libepoxy0 is already the newest version (1.5.10-1).
2025-03-18T16:41:32.326Z libevdev2 is already the newest version (1.13.0+dfsg-1).
2025-03-18T16:41:32.327Z libgles2 is already the newest version (1.6.0-1).
2025-03-18T16:41:32.328Z libglx0 is already the newest version (1.6.0-1).
2025-03-18T16:41:32.329Z libgstreamer-gl1.0-0 is already the newest version (1.22.0-3+deb12u4).
2025-03-18T16:41:32.330Z libgstreamer-plugins-base1.0-0 is already the newest version (1.22.0-3+deb12u4).
2025-03-18T16:41:32.330Z libgstreamer1.0-0 is already the newest version (1.22.0-2+deb12u1).
2025-03-18T16:41:32.332Z libgtk-4-1 is already the newest version (4.8.3+ds-2+deb12u1).
2025-03-18T16:41:32.332Z libgudev-1.0-0 is already the newest version (237-2).
2025-03-18T16:41:32.333Z libharfbuzz-icu0 is already the newest version (6.0.0+dfsg-3).
2025-03-18T16:41:32.334Z libhyphen0 is already the newest version (2.8.8-7).
2025-03-18T16:41:32.335Z libicu72 is already the newest version (72.1-3).
2025-03-18T16:41:32.336Z libjpeg62-turbo is already the newest version (1:2.1.5-2).
2025-03-18T16:41:32.337Z liblcms2-2 is already the newest version (2.14-2).
2025-03-18T16:41:32.339Z libmanette-0.2-0 is already the newest version (0.2.6-3+b1).
2025-03-18T16:41:32.340Z libnotify4 is already the newest version (0.8.1-1).
2025-03-18T16:41:32.342Z libopengl0 is already the newest version (1.6.0-1).
2025-03-18T16:41:32.346Z libopenjp2-7 is already the newest version (2.5.0-2+deb12u1).
2025-03-18T16:41:32.350Z libopus0 is already the newest version (1.3.1-3).
2025-03-18T16:41:32.352Z libpng16-16 is already the newest version (1.6.39-2).
2025-03-18T16:41:32.353Z libproxy1v5 is already the newest version (0.4.18-1.2).
2025-03-18T16:41:32.353Z libsecret-1-0 is already the newest version (0.20.5-3).
2025-03-18T16:41:32.354Z libwayland-client0 is already the newest version (1.21.0-1).
2025-03-18T16:41:32.355Z libwayland-egl1 is already the newest version (1.21.0-1).
2025-03-18T16:41:32.356Z libwayland-server0 is already the newest version (1.21.0-1).
2025-03-18T16:41:32.357Z libwebp7 is already the newest version (1.2.4-0.2+deb12u1).
2025-03-18T16:41:32.358Z libwebpdemux2 is already the newest version (1.2.4-0.2+deb12u1).
2025-03-18T16:41:32.359Z libwoff1 is already the newest version (1.0.2-2).
2025-03-18T16:41:32.360Z libxml2 is already the newest version (2.9.14+dfsg-1.3~deb12u1).
2025-03-18T16:41:32.361Z libxslt1.1 is already the newest version (1.1.35-1).
2025-03-18T16:41:32.361Z libatomic1 is already the newest version (12.2.0-14).
2025-03-18T16:41:32.362Z libevent-2.1-7 is already the newest version (2.1.12-stable-8).
2025-03-18T16:41:32.363Z libavif15 is already the newest version (0.11.1-1).
2025-03-18T16:41:32.364Z xvfb is already the newest version (2:21.1.7-3+deb12u9).
2025-03-18T16:41:32.365Z fonts-noto-color-emoji is already the newest version (2.042-0+deb12u1).
2025-03-18T16:41:32.366Z fonts-unifont is already the newest version (1:15.0.01-2).
2025-03-18T16:41:32.367Z xfonts-scalable is already the newest version (1:1.0.3-1.3).
2025-03-18T16:41:32.368Z fonts-liberation is already the newest version (1:1.07.4-11).
2025-03-18T16:41:32.369Z fonts-ipafont-gothic is already the newest version (00303-23).
2025-03-18T16:41:32.369Z fonts-wqy-zenhei is already the newest version (0.9.45-8).
2025-03-18T16:41:32.370Z fonts-tlwg-loma-otf is already the newest version (1:0.7.3-1).
2025-03-18T16:41:32.371Z fonts-freefont-ttf is already the newest version (20120503-10).
2025-03-18T16:41:32.424Z The following packages will be upgraded:
2025-03-18T16:41:32.425Z   libfreetype6
2025-03-18T16:41:32.426Z 1 upgraded, 0 newly installed, 0 to remove and 17 not upgraded.
2025-03-18T16:41:32.429Z Need to get 398 kB of archives.
2025-03-18T16:41:32.430Z After this operation, 0 B of additional disk space will be used.
2025-03-18T16:41:32.435Z 
0% [Working]
            
Get:1 http://deb.debian.org/debian-security bookworm-security/main amd64 libfreetype6 amd64 2.12.1+dfsg-5+deb12u4 [398 kB]
2025-03-18T16:41:32.439Z 
0% [1 libfreetype6 0 B/398 kB 0%]
                                 
100% [Working]
              
Fetched 398 kB in 0s (26.2 MB/s)
2025-03-18T16:41:32.671Z debconf: delaying package configuration, since apt-utils is not installed
2025-03-18T16:41:33.032Z (Reading database ... 
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 31320 files and directories currently installed.)
2025-03-18T16:41:33.034Z Preparing to unpack .../libfreetype6_2.12.1+dfsg-5+deb12u4_amd64.deb ...
2025-03-18T16:41:33.059Z Unpacking libfreetype6:amd64 (2.12.1+dfsg-5+deb12u4) over (2.12.1+dfsg-5+deb12u3) ...
2025-03-18T16:41:33.129Z Setting up libfreetype6:amd64 (2.12.1+dfsg-5+deb12u4) ...
2025-03-18T16:41:33.141Z Processing triggers for libc-bin (2.36-9+deb12u9) ...
2025-03-18T16:41:34.542Z [apify] INFO  Initializing Actor...
2025-03-18T16:41:34.545Z [apify] INFO  System info ({"apify_sdk_version": "2.4.0", "apify_client_version": "1.9.2", "crawlee_version": "0.6.3", "python_version": "3.13.2", "os": "linux"})
2025-03-18T16:41:34.693Z Processing country: BR
2025-03-18T16:41:34.694Z Attempt 1/3 with smartproxy for BR
2025-03-18T16:41:34.696Z Attempting crawl with smartproxy proxy for https://apuestas.guru/sports/librabet/
2025-03-18T16:41:34.697Z Starting crawl for desktop version of https://apuestas.guru/sports/librabet/ in BR using smartproxy proxy
2025-03-18T16:41:34.795Z Starting crawl for mobile version of https://apuestas.guru/sports/librabet/ in BR using smartproxy proxy
2025-03-18T16:41:35.369Z [INIT].... → Crawl4AI 0.5.0.post4
2025-03-18T16:41:35.386Z [INIT].... → Crawl4AI 0.5.0.post4
2025-03-18T16:41:38.775Z [CONSOLE]. ℹ Console: Failed to load resource: the server responded with a status of 407 ()
2025-03-18T16:41:38.974Z [CONSOLE]. ℹ Console: Failed to load resource: the server responded with a status of 407 ()
2025-03-18T16:41:38.978Z [ERROR]... × https://apuestas.guru/sports/librabet/... | Error:
2025-03-18T16:41:38.979Z ┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
2025-03-18T16:41:38.981Z │ × Unexpected error in _crawl_web at line 579 in _crawl_web (../../local/lib/python3.13/site-                          │
2025-03-18T16:41:38.981Z │ packages/crawl4ai/async_crawler_strategy.py):                                                                         │
2025-03-18T16:41:38.982Z │   Error: Failed on navigating ACS-GOTO:                                                                               │
2025-03-18T16:41:38.983Z │   Page.goto: net::ERR_HTTP_RESPONSE_CODE_FAILURE at https://apuestas.guru/sports/librabet/                            │
2025-03-18T16:41:38.984Z │   Call log:                                                                                                           │
2025-03-18T16:41:38.985Z │   - navigating to "https://apuestas.guru/sports/librabet/", waiting until "networkidle"                               │
2025-03-18T16:41:38.986Z │                                                                                                                       │
2025-03-18T16:41:38.987Z │                                                                                                                       │
2025-03-18T16:41:38.988Z │   Code context:                                                                                                       │
2025-03-18T16:41:38.989Z │   574                       response = await page.goto(                                                               │
2025-03-18T16:41:38.990Z │   575                           url, wait_until=config.wait_until, timeout=config.page_timeout                        │
2025-03-18T16:41:38.991Z │   576                       )                                                                                         │
2025-03-18T16:41:38.992Z │   577                       redirected_url = page.url                                                                 │
2025-03-18T16:41:38.993Z │   578                   except Error as e:                                                                            │
2025-03-18T16:41:38.994Z │   579 →                     raise RuntimeError(f"Failed on navigating ACS-GOTO:\n{str(e)}")                           │
2025-03-18T16:41:38.995Z │   580                                                                                                                 │
2025-03-18T16:41:38.996Z │   581                   await self.execute_hook(                                                                      │
2025-03-18T16:41:38.997Z │   582                       "after_goto", page, context=context, url=url, response=response, config=config            │
2025-03-18T16:41:38.998Z │   583                   )                                                                                             │
2025-03-18T16:41:38.999Z │   584                                                                                                                 │
2025-03-18T16:41:39.000Z └───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
2025-03-18T16:41:39.001Z 
2025-03-18T16:41:39.003Z Failed to crawl https://apuestas.guru/sports/librabet/: Unexpected error in _crawl_web at line 579 in _crawl_web (../../local/lib/python3.13/site-packages/crawl4ai/async_crawler_strategy.py):
2025-03-18T16:41:39.004Z Error: Failed on navigating ACS-GOTO:
2025-03-18T16:41:39.005Z Page.goto: net::ERR_HTTP_RESPONSE_CODE_FAILURE at https://apuestas.guru/sports/librabet/
2025-03-18T16:41:39.006Z Call log:
2025-03-18T16:41:39.008Z   - navigating to "https://apuestas.guru/sports/librabet/", waiting until "networkidle"
2025-03-18T16:41:39.009Z 
2025-03-18T16:41:39.010Z 
2025-03-18T16:41:39.010Z Code context:
2025-03-18T16:41:39.011Z  574                       response = await page.goto(
2025-03-18T16:41:39.012Z  575                           url, wait_until=config.wait_until, timeout=config.page_timeout
2025-03-18T16:41:39.013Z  576                       )
2025-03-18T16:41:39.014Z  577                       redirected_url = page.url
2025-03-18T16:41:39.015Z  578                   except Error as e:
2025-03-18T16:41:39.017Z  579 →                     raise RuntimeError(f"Failed on navigating ACS-GOTO:\n{str(e)}")
2025-03-18T16:41:39.018Z  580
2025-03-18T16:41:39.019Z  581                   await self.execute_hook(
2025-03-18T16:41:39.020Z  582                       "after_goto", page, context=context, url=url, response=response, config=config
2025-03-18T16:41:39.021Z  583                   )
2025-03-18T16:41:39.022Z  584
2025-03-18T16:41:39.081Z [ERROR]... × https://apuestas.guru/sports/librabet/... | Error:
2025-03-18T16:41:39.083Z ┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
2025-03-18T16:41:39.084Z │ × Unexpected error in _crawl_web at line 579 in _crawl_web (../../local/lib/python3.13/site-                          │
2025-03-18T16:41:39.084Z │ packages/crawl4ai/async_crawler_strategy.py):                                                                         │
2025-03-18T16:41:39.085Z │   Error: Failed on navigating ACS-GOTO:                                                                               │
2025-03-18T16:41:39.086Z │   Page.goto: net::ERR_HTTP_RESPONSE_CODE_FAILURE at https://apuestas.guru/sports/librabet/                            │
2025-03-18T16:41:39.088Z │   Call log:                                                                                                           │
2025-03-18T16:41:39.089Z │   - navigating to "https://apuestas.guru/sports/librabet/", waiting until "networkidle"                               │
2025-03-18T16:41:39.090Z │                                                                                                                       │
2025-03-18T16:41:39.091Z │                                                                                                                       │
2025-03-18T16:41:39.092Z │   Code context:                                                                                                       │
2025-03-18T16:41:39.093Z │   574                       response = await page.goto(                                                               │
2025-03-18T16:41:39.094Z │   575                           url, wait_until=config.wait_until, timeout=config.page_timeout                        │
2025-03-18T16:41:39.095Z │   576                       )                                                                                         │
2025-03-18T16:41:39.096Z │   577                       redirected_url = page.url                                                                 │
2025-03-18T16:41:39.097Z │   578                   except Error as e:                                                                            │
2025-03-18T16:41:39.098Z │   579 →                     raise RuntimeError(f"Failed on navigating ACS-GOTO:\n{str(e)}")                           │
2025-03-18T16:41:39.099Z │   580                                                                                                                 │
2025-03-18T16:41:39.100Z │   581                   await self.execute_hook(                                                                      │
2025-03-18T16:41:39.101Z │   582                       "after_goto", page, context=context, url=url, response=response, config=config            │
2025-03-18T16:41:39.102Z │   583                   )                                                                                             │
2025-03-18T16:41:39.103Z │   584                                                                                                                 │
2025-03-18T16:41:39.103Z └───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
2025-03-18T16:41:39.105Z 
2025-03-18T16:41:39.106Z Failed to crawl https://apuestas.guru/sports/librabet/: Unexpected error in _crawl_web at line 579 in _crawl_web (../../local/lib/python3.13/site-packages/crawl4ai/async_crawler_strategy.py):
2025-03-18T16:41:39.107Z Error: Failed on navigating ACS-GOTO:
2025-03-18T16:41:39.108Z Page.goto: net::ERR_HTTP_RESPONSE_CODE_FAILURE at https://apuestas.guru/sports/librabet/
2025-03-18T16:41:39.109Z Call log:
2025-03-18T16:41:39.110Z   - navigating to "https://apuestas.guru/sports/librabet/", waiting until "networkidle"
2025-03-18T16:41:39.111Z 
2025-03-18T16:41:39.112Z 
2025-03-18T16:41:39.113Z Code context:
2025-03-18T16:41:39.114Z  574                       response = await page.goto(
2025-03-18T16:41:39.115Z  575                           url, wait_until=config.wait_until, timeout=config.page_timeout
2025-03-18T16:41:39.116Z  576                       )
2025-03-18T16:41:39.117Z  577                       redirected_url = page.url
2025-03-18T16:41:39.118Z  578                   except Error as e:
2025-03-18T16:41:39.119Z  579 →                     raise RuntimeError(f"Failed on navigating ACS-GOTO:\n{str(e)}")
2025-03-18T16:41:39.120Z  580
2025-03-18T16:41:39.121Z  581                   await self.execute_hook(
2025-03-18T16:41:39.122Z  582                       "after_goto", page, context=context, url=url, response=response, config=config
2025-03-18T16:41:39.123Z  583                   )
2025-03-18T16:41:39.124Z  584
2025-03-18T16:41:39.499Z Attempt 1 failed for smartproxy
2025-03-18T16:41:39.501Z Attempt 2/3 with smartproxy for BR
2025-03-18T16:41:39.501Z Attempting crawl with smartproxy proxy for https://apuestas.guru/sports/librabet/
2025-03-18T16:41:39.503Z Starting crawl for desktop version of https://apuestas.guru/sports/librabet/ in BR using smartproxy proxy
2025-03-18T16:41:39.636Z Starting crawl for mobile version of https://apuestas.guru/sports/librabet/ in BR using smartproxy proxy
2025-03-18T16:41:41.073Z [INIT].... → Crawl4AI 0.5.0.post4
2025-03-18T16:41:41.272Z [INIT].... → Crawl4AI 0.5.0.post4
2025-03-18T16:41:45.204Z [CONSOLE]. ℹ Console: Failed to load resource: the server responded with a status of 407 ()
2025-03-18T16:41:45.474Z [ERROR]... × https://apuestas.guru/sports/librabet/... | Error:
2025-03-18T16:41:45.475Z ┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
2025-03-18T16:41:45.477Z │ × Unexpected error in _crawl_web at line 579 in _crawl_web (../../local/lib/python3.13/site-                          │
2025-03-18T16:41:45.478Z │ packages/crawl4ai/async_crawler_strategy.py):                                                                         │
2025-03-18T16:41:45.479Z │   Error: Failed on navigating ACS-GOTO:                                                                               │
2025-03-18T16:41:45.480Z │   Page.goto: net::ERR_HTTP_RESPONSE_CODE_FAILURE at https://apuestas.guru/sports/librabet/                            │
2025-03-18T16:41:45.481Z │   Call log:                                                                                                           │
2025-03-18T16:41:45.482Z │   - navigating to "https://apuestas.guru/sports/librabet/", waiting until "networkidle"                               │
2025-03-18T16:41:45.483Z │                                                                                                                       │
2025-03-18T16:41:45.484Z │                                                                                                                       │
2025-03-18T16:41:45.487Z │   Code context:                                                                                                       │
2025-03-18T16:41:45.490Z │   574                       response = await page.goto(                                                               │
2025-03-18T16:41:45.491Z │   575                           url, wait_until=config.wait_until, timeout=config.page_timeout                        │
2025-03-18T16:41:45.492Z │   576                       )                                                                                         │
2025-03-18T16:41:45.493Z │   577                       redirected_url = page.url                                                                 │
2025-03-18T16:41:45.494Z │   578                   except Error as e:                                                                            │
2025-03-18T16:41:45.495Z │   579 →                     raise RuntimeError(f"Failed on navigating ACS-GOTO:\n{str(e)}")                           │
2025-03-18T16:41:45.496Z │   580                                                                                                                 │
2025-03-18T16:41:45.497Z │   581                   await self.execute_hook(                                                                      │
2025-03-18T16:41:45.498Z │   582                       "after_goto", page, context=context, url=url, response=response, config=config            │
2025-03-18T16:41:45.498Z │   583                   )                                                                                             │
2025-03-18T16:41:45.499Z │   584                                                                                                                 │
2025-03-18T16:41:45.500Z └───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
2025-03-18T16:41:45.501Z 
2025-03-18T16:41:45.502Z Failed to crawl https://apuestas.guru/sports/librabet/: Unexpected error in _crawl_web at line 579 in _crawl_web (../../local/lib/python3.13/site-packages/crawl4ai/async_crawler_strategy.py):
2025-03-18T16:41:45.503Z Error: Failed on navigating ACS-GOTO:
2025-03-18T16:41:45.504Z Page.goto: net::ERR_HTTP_RESPONSE_CODE_FAILURE at https://apuestas.guru/sports/librabet/
2025-03-18T16:41:45.505Z Call log:
2025-03-18T16:41:45.505Z   - navigating to "https://apuestas.guru/sports/librabet/", waiting until "networkidle"
2025-03-18T16:41:45.506Z 
2025-03-18T16:41:45.508Z 
2025-03-18T16:41:45.509Z Code context:
2025-03-18T16:41:45.510Z  574                       response = await page.goto(
2025-03-18T16:41:45.511Z  575                           url, wait_until=config.wait_until, timeout=config.page_timeout
2025-03-18T16:41:45.512Z  576                       )
2025-03-18T16:41:45.513Z  577                       redirected_url = page.url
2025-03-18T16:41:45.513Z  578                   except Error as e:
2025-03-18T16:41:45.514Z  579 →                     raise RuntimeError(f"Failed on navigating ACS-GOTO:\n{str(e)}")
2025-03-18T16:41:45.515Z  580
2025-03-18T16:41:45.516Z  581                   await self.execute_hook(
2025-03-18T16:41:45.517Z  582                       "after_goto", page, context=context, url=url, response=response, config=config
2025-03-18T16:41:45.518Z  583                   )
2025-03-18T16:41:45.519Z  584
2025-03-18T16:41:45.570Z [CONSOLE]. ℹ Console: Failed to load resource: the server responded with a status of 407 ()
2025-03-18T16:41:45.573Z [ERROR]... × https://apuestas.guru/sports/librabet/... | Error:
2025-03-18T16:41:45.574Z ┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
2025-03-18T16:41:45.575Z │ × Unexpected error in _crawl_web at line 579 in _crawl_web (../../local/lib/python3.13/site-                          │
2025-03-18T16:41:45.576Z │ packages/crawl4ai/async_crawler_strategy.py):                                                                         │
2025-03-18T16:41:45.576Z │   Error: Failed on navigating ACS-GOTO:                                                                               │
2025-03-18T16:41:45.579Z │   Page.goto: net::ERR_HTTP_RESPONSE_CODE_FAILURE at https://apuestas.guru/sports/librabet/                            │
2025-03-18T16:41:45.580Z │   Call log:                                                                                                           │
2025-03-18T16:41:45.581Z │   - navigating to "https://apuestas.guru/sports/librabet/", waiting until "networkidle"                               │
2025-03-18T16:41:45.582Z │                                                                                                                       │
2025-03-18T16:41:45.583Z │                                                                                                                       │
2025-03-18T16:41:45.583Z │   Code context:                                                                                                       │
2025-03-18T16:41:45.584Z │   574                       response = await page.goto(                                                               │
2025-03-18T16:41:45.585Z │   575                           url, wait_until=config.wait_until, timeout=config.page_timeout                        │
2025-03-18T16:41:45.586Z │   576                       )                                                                                         │
2025-03-18T16:41:45.587Z │   577                       redirected_url = page.url                                                                 │
2025-03-18T16:41:45.588Z │   578                   except Error as e:                                                                            │
2025-03-18T16:41:45.589Z │   579 →                     raise RuntimeError(f"Failed on navigating ACS-GOTO:\n{str(e)}")                           │
2025-03-18T16:41:45.590Z │   580                                                                                                                 │
2025-03-18T16:41:45.591Z │   581                   await self.execute_hook(                                                                      │
2025-03-18T16:41:45.592Z │   582                       "after_goto", page, context=context, url=url, response=response, config=config            │
2025-03-18T16:41:45.593Z │   583                   )                                                                                             │
2025-03-18T16:41:45.594Z │   584                                                                                                                 │
2025-03-18T16:41:45.595Z └───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
2025-03-18T16:41:45.596Z 
2025-03-18T16:41:45.597Z Failed to crawl https://apuestas.guru/sports/librabet/: Unexpected error in _crawl_web at line 579 in _crawl_web (../../local/lib/python3.13/site-packages/crawl4ai/async_crawler_strategy.py):
2025-03-18T16:41:45.597Z Error: Failed on navigating ACS-GOTO:
2025-03-18T16:41:45.598Z Page.goto: net::ERR_HTTP_RESPONSE_CODE_FAILURE at https://apuestas.guru/sports/librabet/
2025-03-18T16:41:45.599Z Call log:
2025-03-18T16:41:45.600Z   - navigating to "https://apuestas.guru/sports/librabet/", waiting until "networkidle"
2025-03-18T16:41:45.601Z 
2025-03-18T16:41:45.602Z 
2025-03-18T16:41:45.602Z Code context:
2025-03-18T16:41:45.603Z  574                       response = await page.goto(
2025-03-18T16:41:45.604Z  575                           url, wait_until=config.wait_until, timeout=config.page_timeout
2025-03-18T16:41:45.606Z  576                       )
2025-03-18T16:41:45.607Z  577                       redirected_url = page.url
2025-03-18T16:41:45.609Z  578                   except Error as e:
2025-03-18T16:41:45.609Z  579 →                     raise RuntimeError(f"Failed on navigating ACS-GOTO:\n{str(e)}")
2025-03-18T16:41:45.611Z  580
2025-03-18T16:41:45.611Z  581                   await self.execute_hook(
2025-03-18T16:41:45.613Z  582                       "after_goto", page, context=context, url=url, response=response, config=config
2025-03-18T16:41:45.613Z  583                   )
2025-03-18T16:41:45.615Z  584
2025-03-18T16:41:45.895Z Attempt 2 failed for smartproxy
2025-03-18T16:41:45.897Z Attempt 3/3 with smartproxy for BR
2025-03-18T16:41:45.897Z Attempting crawl with smartproxy proxy for https://apuestas.guru/sports/librabet/
2025-03-18T16:41:45.898Z Starting crawl for desktop version of https://apuestas.guru/sports/librabet/ in BR using smartproxy proxy
2025-03-18T16:41:46.017Z Starting crawl for mobile version of https://apuestas.guru/sports/librabet/ in BR using smartproxy proxy
2025-03-18T16:41:47.478Z [INIT].... → Crawl4AI 0.5.0.post4
2025-03-18T16:41:47.723Z [INIT].... → Crawl4AI 0.5.0.post4
2025-03-18T16:41:51.677Z [CONSOLE]. ℹ Console: Failed to load resource: the server responded with a status of 407 ()
2025-03-18T16:41:51.871Z [CONSOLE]. ℹ Console: Failed to load resource: the server responded with a status of 407 ()
2025-03-18T16:41:51.873Z [ERROR]... × https://apuestas.guru/sports/librabet/... | Error:
2025-03-18T16:41:51.874Z ┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
2025-03-18T16:41:51.874Z │ × Unexpected error in _crawl_web at line 579 in _crawl_web (../../local/lib/python3.13/site-                          │
2025-03-18T16:41:51.875Z │ packages/crawl4ai/async_crawler_strategy.py):                                                                         │
2025-03-18T16:41:51.875Z │   Error: Failed on navigating ACS-GOTO:                                                                               │
2025-03-18T16:41:51.876Z │   Page.goto: net::ERR_HTTP_RESPONSE_CODE_FAILURE at https://apuestas.guru/sports/librabet/                            │
2025-03-18T16:41:51.877Z │   Call log:                                                                                                           │
2025-03-18T16:41:51.878Z │   - navigating to "https://apuestas.guru/sports/librabet/", waiting until "networkidle"                               │
2025-03-18T16:41:51.879Z │                                                                                                                       │
2025-03-18T16:41:51.880Z │                                                                                                                       │
2025-03-18T16:41:51.880Z │   Code context:                                                                                                       │
2025-03-18T16:41:51.881Z │   574                       response = await page.goto(                                                               │
2025-03-18T16:41:51.882Z │   575                           url, wait_until=config.wait_until, timeout=config.page_timeout                        │
2025-03-18T16:41:51.882Z │   576                       )                                                                                         │
2025-03-18T16:41:51.883Z │   577                       redirected_url = page.url                                                                 │
2025-03-18T16:41:51.884Z │   578                   except Error as e:                                                                            │
2025-03-18T16:41:51.885Z │   579 →                     raise RuntimeError(f"Failed on navigating ACS-GOTO:\n{str(e)}")                           │
2025-03-18T16:41:51.885Z │   580                                                                                                                 │
2025-03-18T16:41:51.886Z │   581                   await self.execute_hook(                                                                      │
2025-03-18T16:41:51.887Z │   582                       "after_goto", page, context=context, url=url, response=response, config=config            │
2025-03-18T16:41:51.888Z │   583                   )                                                                                             │
2025-03-18T16:41:51.888Z │   584                                                                                                                 │
2025-03-18T16:41:51.889Z └───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
2025-03-18T16:41:51.890Z 
2025-03-18T16:41:51.891Z Failed to crawl https://apuestas.guru/sports/librabet/: Unexpected error in _crawl_web at line 579 in _crawl_web (../../local/lib/python3.13/site-packages/crawl4ai/async_crawler_strategy.py):
2025-03-18T16:41:51.891Z Error: Failed on navigating ACS-GOTO:
2025-03-18T16:41:51.892Z Page.goto: net::ERR_HTTP_RESPONSE_CODE_FAILURE at https://apuestas.guru/sports/librabet/
2025-03-18T16:41:51.892Z Call log:
2025-03-18T16:41:51.893Z   - navigating to "https://apuestas.guru/sports/librabet/", waiting until "networkidle"
2025-03-18T16:41:51.894Z 
2025-03-18T16:41:51.894Z 
2025-03-18T16:41:51.895Z Code context:
2025-03-18T16:41:51.895Z  574                       response = await page.goto(
2025-03-18T16:41:51.896Z  575                           url, wait_until=config.wait_until, timeout=config.page_timeout
2025-03-18T16:41:51.897Z  576                       )
2025-03-18T16:41:51.897Z  577                       redirected_url = page.url
2025-03-18T16:41:51.898Z  578                   except Error as e:
2025-03-18T16:41:51.898Z  579 →                     raise RuntimeError(f"Failed on navigating ACS-GOTO:\n{str(e)}")
2025-03-18T16:41:51.899Z  580
2025-03-18T16:41:51.899Z  581                   await self.execute_hook(
2025-03-18T16:41:51.900Z  582                       "after_goto", page, context=context, url=url, response=response, config=config
2025-03-18T16:41:51.901Z  583                   )
2025-03-18T16:41:51.901Z  584
2025-03-18T16:41:51.902Z [ERROR]... × https://apuestas.guru/sports/librabet/... | Error:
2025-03-18T16:41:51.902Z ┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
2025-03-18T16:41:51.903Z │ × Unexpected error in _crawl_web at line 579 in _crawl_web (../../local/lib/python3.13/site-                          │
2025-03-18T16:41:51.903Z │ packages/crawl4ai/async_crawler_strategy.py):                                                                         │
2025-03-18T16:41:51.904Z │   Error: Failed on navigating ACS-GOTO:                                                                               │
2025-03-18T16:41:51.904Z │   Page.goto: net::ERR_HTTP_RESPONSE_CODE_FAILURE at https://apuestas.guru/sports/librabet/                            │
2025-03-18T16:41:51.905Z │   Call log:                                                                                                           │
2025-03-18T16:41:51.905Z │   - navigating to "https://apuestas.guru/sports/librabet/", waiting until "networkidle"                               │
2025-03-18T16:41:51.906Z │                                                                                                                       │
2025-03-18T16:41:51.907Z │                                                                                                                       │
2025-03-18T16:41:51.907Z │   Code context:                                                                                                       │
2025-03-18T16:41:51.909Z │   574                       response = await page.goto(                                                               │
2025-03-18T16:41:51.909Z │   575                           url, wait_until=config.wait_until, timeout=config.page_timeout                        │
2025-03-18T16:41:51.910Z │   576                       )                                                                                         │
2025-03-18T16:41:51.910Z │   577                       redirected_url = page.url                                                                 │
2025-03-18T16:41:51.911Z │   578                   except Error as e:                                                                            │
2025-03-18T16:41:51.911Z │   579 →                     raise RuntimeError(f"Failed on navigating ACS-GOTO:\n{str(e)}")                           │
2025-03-18T16:41:51.912Z │   580                                                                                                                 │
2025-03-18T16:41:51.912Z │   581                   await self.execute_hook(                                                                      │
2025-03-18T16:41:51.913Z │   582                       "after_goto", page, context=context, url=url, response=response, config=config            │
2025-03-18T16:41:51.913Z │   583                   )                                                                                             │
2025-03-18T16:41:51.914Z │   584                                                                                                                 │
2025-03-18T16:41:51.915Z └───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
2025-03-18T16:41:51.915Z 
2025-03-18T16:41:51.916Z Failed to crawl https://apuestas.guru/sports/librabet/: Unexpected error in _crawl_web at line 579 in _crawl_web (../../local/lib/python3.13/site-packages/crawl4ai/async_crawler_strategy.py):
2025-03-18T16:41:51.916Z Error: Failed on navigating ACS-GOTO:
2025-03-18T16:41:51.917Z Page.goto: net::ERR_HTTP_RESPONSE_CODE_FAILURE at https://apuestas.guru/sports/librabet/
2025-03-18T16:41:51.917Z Call log:
2025-03-18T16:41:51.918Z   - navigating to "https://apuestas.guru/sports/librabet/", waiting until "networkidle"
2025-03-18T16:41:51.918Z 
2025-03-18T16:41:51.919Z 
2025-03-18T16:41:51.919Z Code context:
2025-03-18T16:41:51.920Z  574                       response = await page.goto(
2025-03-18T16:41:51.921Z  575                           url, wait_until=config.wait_until, timeout=config.page_timeout
2025-03-18T16:41:51.921Z  576                       )
2025-03-18T16:41:51.922Z  577                       redirected_url = page.url
2025-03-18T16:41:51.922Z  578                   except Error as e:
2025-03-18T16:41:51.923Z  579 →                     raise RuntimeError(f"Failed on navigating ACS-GOTO:\n{str(e)}")
2025-03-18T16:41:51.924Z  580
2025-03-18T16:41:51.924Z  581                   await self.execute_hook(
2025-03-18T16:41:51.925Z  582                       "after_goto", page, context=context, url=url, response=response, config=config
2025-03-18T16:41:51.925Z  583                   )
2025-03-18T16:41:51.926Z  584
2025-03-18T16:41:52.287Z Attempt 3 failed for smartproxy
2025-03-18T16:41:52.288Z Attempt 1/3 with goproxies for BR
2025-03-18T16:41:52.289Z Attempting crawl with goproxies proxy for https://apuestas.guru/sports/librabet/
2025-03-18T16:41:52.290Z Starting crawl for desktop version of https://apuestas.guru/sports/librabet/ in BR using goproxies proxy
2025-03-18T16:41:52.423Z Starting crawl for mobile version of https://apuestas.guru/sports/librabet/ in BR using goproxies proxy
2025-03-18T16:41:53.968Z [INIT].... → Crawl4AI 0.5.0.post4
2025-03-18T16:41:54.077Z [INIT].... → Crawl4AI 0.5.0.post4
2025-03-18T16:41:55.589Z [ERROR]... × https://apuestas.guru/sports/librabet/... | Error:
2025-03-18T16:41:55.590Z ┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
2025-03-18T16:41:55.591Z │ × Unexpected error in _crawl_web at line 579 in _crawl_web (../../local/lib/python3.13/site-                          │
2025-03-18T16:41:55.592Z │ packages/crawl4ai/async_crawler_strategy.py):                                                                         │
2025-03-18T16:41:55.593Z │   Error: Failed on navigating ACS-GOTO:                                                                               │
2025-03-18T16:41:55.594Z │   Page.goto: net::ERR_TUNNEL_CONNECTION_FAILED at https://apuestas.guru/sports/librabet/                              │
2025-03-18T16:41:55.595Z │   Call log:                                                                                                           │
2025-03-18T16:41:55.596Z │   - navigating to "https://apuestas.guru/sports/librabet/", waiting until "networkidle"                               │
2025-03-18T16:41:55.597Z │                                                                                                                       │
2025-03-18T16:41:55.598Z │                                                                                                                       │
2025-03-18T16:41:55.599Z │   Code context:                                                                                                       │
2025-03-18T16:41:55.600Z │   574                       response = await page.goto(                                                               │
2025-03-18T16:41:55.601Z │   575                           url, wait_until=config.wait_until, timeout=config.page_timeout                        │
2025-03-18T16:41:55.602Z │   576                       )                                                                                         │
2025-03-18T16:41:55.603Z │   577                       redirected_url = page.url                                                                 │
2025-03-18T16:41:55.604Z │   578                   except Error as e:                                                                            │
2025-03-18T16:41:55.605Z │   579 →                     raise RuntimeError(f"Failed on navigating ACS-GOTO:\n{str(e)}")                           │
2025-03-18T16:41:55.606Z │   580                                                                                                                 │
2025-03-18T16:41:55.607Z │   581                   await self.execute_hook(                                                                      │
2025-03-18T16:41:55.608Z │   582                       "after_goto", page, context=context, url=url, response=response, config=config            │
2025-03-18T16:41:55.609Z │   583                   )                                                                                             │
2025-03-18T16:41:55.610Z │   584                                                                                                                 │
2025-03-18T16:41:55.611Z └───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
2025-03-18T16:41:55.612Z 
2025-03-18T16:41:55.613Z Failed to crawl https://apuestas.guru/sports/librabet/: Unexpected error in _crawl_web at line 579 in _crawl_web (../../local/lib/python3.13/site-packages/crawl4ai/async_crawler_strategy.py):
2025-03-18T16:41:55.614Z Error: Failed on navigating ACS-GOTO:
2025-03-18T16:41:55.615Z Page.goto: net::ERR_TUNNEL_CONNECTION_FAILED at https://apuestas.guru/sports/librabet/
2025-03-18T16:41:55.616Z Call log:
2025-03-18T16:41:55.617Z   - navigating to "https://apuestas.guru/sports/librabet/", waiting until "networkidle"
2025-03-18T16:41:55.618Z 
2025-03-18T16:41:55.619Z 
2025-03-18T16:41:55.620Z Code context:
2025-03-18T16:41:55.620Z  574                       response = await page.goto(
2025-03-18T16:41:55.621Z  575                           url, wait_until=config.wait_until, timeout=config.page_timeout
2025-03-18T16:41:55.622Z  576                       )
2025-03-18T16:41:55.623Z  577                       redirected_url = page.url
2025-03-18T16:41:55.624Z  578                   except Error as e:
2025-03-18T16:41:55.625Z  579 →                     raise RuntimeError(f"Failed on navigating ACS-GOTO:\n{str(e)}")
2025-03-18T16:41:55.626Z  580
2025-03-18T16:41:55.627Z  581                   await self.execute_hook(
2025-03-18T16:41:55.628Z  582                       "after_goto", page, context=context, url=url, response=response, config=config
2025-03-18T16:41:55.629Z  583                   )
2025-03-18T16:41:55.630Z  584
2025-03-18T16:41:55.878Z [ERROR]... × https://apuestas.guru/sports/librabet/... | Error:
2025-03-18T16:41:55.880Z ┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
2025-03-18T16:41:55.881Z │ × Unexpected error in _crawl_web at line 579 in _crawl_web (../../local/lib/python3.13/site-                          │
2025-03-18T16:41:55.882Z │ packages/crawl4ai/async_crawler_strategy.py):                                                                         │
2025-03-18T16:41:55.883Z │   Error: Failed on navigating ACS-GOTO:                                                                               │
2025-03-18T16:41:55.886Z │   Page.goto: net::ERR_TUNNEL_CONNECTION_FAILED at https://apuestas.guru/sports/librabet/                              │
2025-03-18T16:41:55.887Z │   Call log:                                                                                                           │
2025-03-18T16:41:55.887Z │   - navigating to "https://apuestas.guru/sports/librabet/", waiting until "networkidle"                               │
2025-03-18T16:41:55.888Z │                                                                                                                       │
2025-03-18T16:41:55.890Z │                                                                                                                       │
2025-03-18T16:41:55.890Z │   Code context:                                                                                                       │
2025-03-18T16:41:55.891Z │   574                       response = await page.goto(                                                               │
2025-03-18T16:41:55.892Z │   575                           url, wait_until=config.wait_until, timeout=config.page_timeout                        │
2025-03-18T16:41:55.893Z │   576                       )                                                                                         │
2025-03-18T16:41:55.894Z │   577                       redirected_url = page.url                                                                 │
2025-03-18T16:41:55.895Z │   578                   except Error as e:                                                                            │
2025-03-18T16:41:55.896Z │   579 →                     raise RuntimeError(f"Failed on navigating ACS-GOTO:\n{str(e)}")                           │
2025-03-18T16:41:55.897Z │   580                                                                                                                 │
2025-03-18T16:41:55.898Z │   581                   await self.execute_hook(                                                                      │
2025-03-18T16:41:55.899Z │   582                       "after_goto", page, context=context, url=url, response=response, config=config            │
2025-03-18T16:41:55.900Z │   583                   )                                                                                             │
2025-03-18T16:41:55.901Z │   584                                                                                                                 │
2025-03-18T16:41:55.902Z └───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
2025-03-18T16:41:55.903Z 
2025-03-18T16:41:55.904Z Failed to crawl https://apuestas.guru/sports/librabet/: Unexpected error in _crawl_web at line 579 in _crawl_web (../../local/lib/python3.13/site-packages/crawl4ai/async_crawler_strategy.py):
2025-03-18T16:41:55.905Z Error: Failed on navigating ACS-GOTO:
2025-03-18T16:41:55.906Z Page.goto: net::ERR_TUNNEL_CONNECTION_FAILED at https://apuestas.guru/sports/librabet/
2025-03-18T16:41:55.907Z Call log:
2025-03-18T16:41:55.908Z   - navigating to "https://apuestas.guru/sports/librabet/", waiting until "networkidle"
2025-03-18T16:41:55.909Z 
2025-03-18T16:41:55.910Z 
2025-03-18T16:41:55.911Z Code context:
2025-03-18T16:41:55.912Z  574                       response = await page.goto(
2025-03-18T16:41:55.913Z  575                           url, wait_until=config.wait_until, timeout=config.page_timeout
2025-03-18T16:41:55.914Z  576                       )
2025-03-18T16:41:55.915Z  577                       redirected_url = page.url
2025-03-18T16:41:55.916Z  578                   except Error as e:
2025-03-18T16:41:55.917Z  579 →                     raise RuntimeError(f"Failed on navigating ACS-GOTO:\n{str(e)}")
2025-03-18T16:41:55.918Z  580
2025-03-18T16:41:55.919Z  581                   await self.execute_hook(
2025-03-18T16:41:55.920Z  582                       "after_goto", page, context=context, url=url, response=response, config=config
2025-03-18T16:41:55.921Z  583                   )
2025-03-18T16:41:55.922Z  584
2025-03-18T16:41:56.195Z Attempt 1 failed for goproxies
2025-03-18T16:41:56.196Z Attempt 2/3 with goproxies for BR
2025-03-18T16:41:56.197Z Attempting crawl with goproxies proxy for https://apuestas.guru/sports/librabet/
2025-03-18T16:41:56.198Z Starting crawl for desktop version of https://apuestas.guru/sports/librabet/ in BR using goproxies proxy
2025-03-18T16:41:56.368Z Starting crawl for mobile version of https://apuestas.guru/sports/librabet/ in BR using goproxies proxy
2025-03-18T16:41:57.774Z [INIT].... → Crawl4AI 0.5.0.post4
2025-03-18T16:41:58.018Z [INIT].... → Crawl4AI 0.5.0.post4
2025-03-18T16:41:59.788Z [ERROR]... × https://apuestas.guru/sports/librabet/... | Error:
2025-03-18T16:41:59.789Z ┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
2025-03-18T16:41:59.790Z │ × Unexpected error in _crawl_web at line 579 in _crawl_web (../../local/lib/python3.13/site-                          │
2025-03-18T16:41:59.791Z │ packages/crawl4ai/async_crawler_strategy.py):                                                                         │
2025-03-18T16:41:59.792Z │   Error: Failed on navigating ACS-GOTO:                                                                               │
2025-03-18T16:41:59.793Z │   Page.goto: net::ERR_TUNNEL_CONNECTION_FAILED at https://apuestas.guru/sports/librabet/                              │
2025-03-18T16:41:59.794Z │   Call log:                                                                                                           │
2025-03-18T16:41:59.795Z │   - navigating to "https://apuestas.guru/sports/librabet/", waiting until "networkidle"                               │
2025-03-18T16:41:59.796Z │                                                                                                                       │
2025-03-18T16:41:59.797Z │                                                                                                                       │
2025-03-18T16:41:59.798Z │   Code context:                                                                                                       │
2025-03-18T16:41:59.799Z │   574                       response = await page.goto(                                                               │
2025-03-18T16:41:59.800Z │   575                           url, wait_until=config.wait_until, timeout=config.page_timeout                        │
2025-03-18T16:41:59.801Z │   576                       )                                                                                         │
2025-03-18T16:41:59.801Z │   577                       redirected_url = page.url                                                                 │
2025-03-18T16:41:59.802Z │   578                   except Error as e:                                                                            │
2025-03-18T16:41:59.803Z │   579 →                     raise RuntimeError(f"Failed on navigating ACS-GOTO:\n{str(e)}")                           │
2025-03-18T16:41:59.804Z │   580                                                                                                                 │
2025-03-18T16:41:59.805Z │   581                   await self.execute_hook(                                                                      │
2025-03-18T16:41:59.806Z │   582                       "after_goto", page, context=context, url=url, response=response, config=config            │
2025-03-18T16:41:59.807Z │   583                   )                                                                                             │
2025-03-18T16:41:59.808Z │   584                                                                                                                 │
2025-03-18T16:41:59.809Z └───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
2025-03-18T16:41:59.810Z 
2025-03-18T16:41:59.811Z Failed to crawl https://apuestas.guru/sports/librabet/: Unexpected error in _crawl_web at line 579 in _crawl_web (../../local/lib/python3.13/site-packages/crawl4ai/async_crawler_strategy.py):
2025-03-18T16:41:59.812Z Error: Failed on navigating ACS-GOTO:
2025-03-18T16:41:59.812Z Page.goto: net::ERR_TUNNEL_CONNECTION_FAILED at https://apuestas.guru/sports/librabet/
2025-03-18T16:41:59.813Z Call log:
2025-03-18T16:41:59.814Z   - navigating to "https://apuestas.guru/sports/librabet/", waiting until "networkidle"
2025-03-18T16:41:59.815Z 
2025-03-18T16:41:59.816Z 
2025-03-18T16:41:59.817Z Code context:
2025-03-18T16:41:59.818Z  574                       response = await page.goto(
2025-03-18T16:41:59.818Z  575                           url, wait_until=config.wait_until, timeout=config.page_timeout
2025-03-18T16:41:59.819Z  576                       )
2025-03-18T16:41:59.820Z  577                       redirected_url = page.url
2025-03-18T16:41:59.821Z  578                   except Error as e:
2025-03-18T16:41:59.822Z  579 →                     raise RuntimeError(f"Failed on navigating ACS-GOTO:\n{str(e)}")
2025-03-18T16:41:59.823Z  580
2025-03-18T16:41:59.823Z  581                   await self.execute_hook(
2025-03-18T16:41:59.824Z  582                       "after_goto", page, context=context, url=url, response=response, config=config
2025-03-18T16:41:59.825Z  583                   )
2025-03-18T16:41:59.826Z  584
2025-03-18T16:42:00.176Z [ERROR]... × https://apuestas.guru/sports/librabet/... | Error:
2025-03-18T16:42:00.178Z ┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
2025-03-18T16:42:00.178Z │ × Unexpected error in _crawl_web at line 579 in _crawl_web (../../local/lib/python3.13/site-                          │
2025-03-18T16:42:00.179Z │ packages/crawl4ai/async_crawler_strategy.py):                                                                         │
2025-03-18T16:42:00.180Z │   Error: Failed on navigating ACS-GOTO:                                                                               │
2025-03-18T16:42:00.181Z │   Page.goto: net::ERR_TUNNEL_CONNECTION_FAILED at https://apuestas.guru/sports/librabet/                              │
2025-03-18T16:42:00.182Z │   Call log:                                                                                                           │
2025-03-18T16:42:00.184Z │   - navigating to "https://apuestas.guru/sports/librabet/", waiting until "networkidle"                               │
2025-03-18T16:42:00.185Z │                                                                                                                       │
2025-03-18T16:42:00.186Z │                                                                                                                       │
2025-03-18T16:42:00.187Z │   Code context:                                                                                                       │
2025-03-18T16:42:00.187Z │   574                       response = await page.goto(                                                               │
2025-03-18T16:42:00.188Z │   575                           url, wait_until=config.wait_until, timeout=config.page_timeout                        │
2025-03-18T16:42:00.189Z │   576                       )                                                                                         │
2025-03-18T16:42:00.190Z │   577                       redirected_url = page.url                                                                 │
2025-03-18T16:42:00.191Z │   578                   except Error as e:                                                                            │
2025-03-18T16:42:00.192Z │   579 →                     raise RuntimeError(f"Failed on navigating ACS-GOTO:\n{str(e)}")                           │
2025-03-18T16:42:00.192Z │   580                                                                                                                 │
2025-03-18T16:42:00.193Z │   581                   await self.execute_hook(                                                                      │
2025-03-18T16:42:00.194Z │   582                       "after_goto", page, context=context, url=url, response=response, config=config            │
2025-03-18T16:42:00.195Z │   583                   )                                                                                             │
2025-03-18T16:42:00.196Z │   584                                                                                                                 │
2025-03-18T16:42:00.197Z └───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
2025-03-18T16:42:00.198Z 
2025-03-18T16:42:00.198Z Failed to crawl https://apuestas.guru/sports/librabet/: Unexpected error in _crawl_web at line 579 in _crawl_web (../../local/lib/python3.13/site-packages/crawl4ai/async_crawler_strategy.py):
2025-03-18T16:42:00.199Z Error: Failed on navigating ACS-GOTO:
2025-03-18T16:42:00.200Z Page.goto: net::ERR_TUNNEL_CONNECTION_FAILED at https://apuestas.guru/sports/librabet/
2025-03-18T16:42:00.201Z Call log:
2025-03-18T16:42:00.202Z   - navigating to "https://apuestas.guru/sports/librabet/", waiting until "networkidle"
2025-03-18T16:42:00.203Z 
2025-03-18T16:42:00.203Z 
2025-03-18T16:42:00.204Z Code context:
2025-03-18T16:42:00.205Z  574                       response = await page.goto(
2025-03-18T16:42:00.206Z  575                           url, wait_until=config.wait_until, timeout=config.page_timeout
2025-03-18T16:42:00.207Z  576                       )
2025-03-18T16:42:00.207Z  577                       redirected_url = page.url
2025-03-18T16:42:00.208Z  578                   except Error as e:
2025-03-18T16:42:00.209Z  579 →                     raise RuntimeError(f"Failed on navigating ACS-GOTO:\n{str(e)}")
2025-03-18T16:42:00.210Z  580
2025-03-18T16:42:00.211Z  581                   await self.execute_hook(
2025-03-18T16:42:00.212Z  582                       "after_goto", page, context=context, url=url, response=response, config=config
2025-03-18T16:42:00.213Z  583                   )
2025-03-18T16:42:00.213Z  584
2025-03-18T16:42:00.384Z Attempt 2 failed for goproxies
2025-03-18T16:42:00.386Z Attempt 3/3 with goproxies for BR
2025-03-18T16:42:00.387Z Attempting crawl with goproxies proxy for https://apuestas.guru/sports/librabet/
2025-03-18T16:42:00.388Z Starting crawl for desktop version of https://apuestas.guru/sports/librabet/ in BR using goproxies proxy
2025-03-18T16:42:00.473Z Starting crawl for mobile version of https://apuestas.guru/sports/librabet/ in BR using goproxies proxy
2025-03-18T16:42:01.871Z [INIT].... → Crawl4AI 0.5.0.post4
2025-03-18T16:42:02.181Z [INIT].... → Crawl4AI 0.5.0.post4
2025-03-18T16:42:03.868Z [ERROR]... × https://apuestas.guru/sports/librabet/... | Error:
2025-03-18T16:42:03.871Z ┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
2025-03-18T16:42:03.874Z │ × Unexpected error in _crawl_web at line 579 in _crawl_web (../../local/lib/python3.13/site-                          │
2025-03-18T16:42:03.875Z │ packages/crawl4ai/async_crawler_strategy.py):                                                                         │
2025-03-18T16:42:03.876Z │   Error: Failed on navigating ACS-GOTO:                                                                               │
2025-03-18T16:42:03.877Z │   Page.goto: net::ERR_TUNNEL_CONNECTION_FAILED at https://apuestas.guru/sports/librabet/                              │
2025-03-18T16:42:03.879Z │   Call log:                                                                                                           │
2025-03-18T16:42:03.880Z │   - navigating to "https://apuestas.guru/sports/librabet/", waiting until "networkidle"                               │
2025-03-18T16:42:03.881Z │                                                                                                                       │
2025-03-18T16:42:03.882Z │                                                                                                                       │
2025-03-18T16:42:03.883Z │   Code context:                                                                                                       │
2025-03-18T16:42:03.884Z │   574                       response = await page.goto(                                                               │
2025-03-18T16:42:03.885Z │   575                           url, wait_until=config.wait_until, timeout=config.page_timeout                        │
2025-03-18T16:42:03.886Z │   576                       )                                                                                         │
2025-03-18T16:42:03.887Z │   577                       redirected_url = page.url                                                                 │
2025-03-18T16:42:03.888Z │   578                   except Error as e:                                                                            │
2025-03-18T16:42:03.889Z │   579 →                     raise RuntimeError(f"Failed on navigating ACS-GOTO:\n{str(e)}")                           │
2025-03-18T16:42:03.890Z │   580                                                                                                                 │
2025-03-18T16:42:03.891Z │   581                   await self.execute_hook(                                                                      │
2025-03-18T16:42:03.892Z │   582                       "after_goto", page, context=context, url=url, response=response, config=config            │
2025-03-18T16:42:03.893Z │   583                   )                                                                                             │
2025-03-18T16:42:03.894Z │   584                                                                                                                 │
2025-03-18T16:42:03.895Z └───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
2025-03-18T16:42:03.897Z 
2025-03-18T16:42:03.898Z Failed to crawl https://apuestas.guru/sports/librabet/: Unexpected error in _crawl_web at line 579 in _crawl_web (../../local/lib/python3.13/site-packages/crawl4ai/async_crawler_strategy.py):
2025-03-18T16:42:03.898Z Error: Failed on navigating ACS-GOTO:
2025-03-18T16:42:03.899Z Page.goto: net::ERR_TUNNEL_CONNECTION_FAILED at https://apuestas.guru/sports/librabet/
2025-03-18T16:42:03.900Z Call log:
2025-03-18T16:42:03.901Z   - navigating to "https://apuestas.guru/sports/librabet/", waiting until "networkidle"
2025-03-18T16:42:03.902Z 
2025-03-18T16:42:03.903Z 
2025-03-18T16:42:03.904Z Code context:
2025-03-18T16:42:03.906Z  574                       response = await page.goto(
2025-03-18T16:42:03.907Z  575                           url, wait_until=config.wait_until, timeout=config.page_timeout
2025-03-18T16:42:03.908Z  576                       )
2025-03-18T16:42:03.909Z  577                       redirected_url = page.url
2025-03-18T16:42:03.910Z  578                   except Error as e:
2025-03-18T16:42:03.911Z  579 →                     raise RuntimeError(f"Failed on navigating ACS-GOTO:\n{str(e)}")
2025-03-18T16:42:03.912Z  580
2025-03-18T16:42:03.914Z  581                   await self.execute_hook(
2025-03-18T16:42:03.915Z  582                       "after_goto", page, context=context, url=url, response=response, config=config
2025-03-18T16:42:03.916Z  583                   )
2025-03-18T16:42:03.917Z  584
2025-03-18T16:42:05.285Z [ERROR]... × https://apuestas.guru/sports/librabet/... | Error:
2025-03-18T16:42:05.287Z ┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
2025-03-18T16:42:05.288Z │ × Unexpected error in _crawl_web at line 579 in _crawl_web (../../local/lib/python3.13/site-                          │
2025-03-18T16:42:05.289Z │ packages/crawl4ai/async_crawler_strategy.py):                                                                         │
2025-03-18T16:42:05.291Z │   Error: Failed on navigating ACS-GOTO:                                                                               │
2025-03-18T16:42:05.292Z │   Page.goto: net::ERR_TUNNEL_CONNECTION_FAILED at https://apuestas.guru/sports/librabet/                              │
2025-03-18T16:42:05.293Z │   Call log:                                                                                                           │
2025-03-18T16:42:05.294Z │   - navigating to "https://apuestas.guru/sports/librabet/", waiting until "networkidle"                               │
2025-03-18T16:42:05.295Z │                                                                                                                       │
2025-03-18T16:42:05.296Z │                                                                                                                       │
2025-03-18T16:42:05.297Z │   Code context:                                                                                                       │
2025-03-18T16:42:05.298Z │   574                       response = await page.goto(                                                               │
2025-03-18T16:42:05.298Z │   575                           url, wait_until=config.wait_until, timeout=config.page_timeout                        │
2025-03-18T16:42:05.299Z │   576                       )                                                                                         │
2025-03-18T16:42:05.300Z │   577                       redirected_url = page.url                                                                 │
2025-03-18T16:42:05.301Z │   578                   except Error as e:                                                                            │
2025-03-18T16:42:05.302Z │   579 →                     raise RuntimeError(f"Failed on navigating ACS-GOTO:\n{str(e)}")                           │
2025-03-18T16:42:05.303Z │   580                                                                                                                 │
2025-03-18T16:42:05.304Z │   581                   await self.execute_hook(                                                                      │
2025-03-18T16:42:05.305Z │   582                       "after_goto", page, context=context, url=url, response=response, config=config            │
2025-03-18T16:42:05.306Z │   583                   )                                                                                             │
2025-03-18T16:42:05.307Z │   584                                                                                                                 │
2025-03-18T16:42:05.308Z └───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
2025-03-18T16:42:05.309Z 
2025-03-18T16:42:05.311Z Failed to crawl https://apuestas.guru/sports/librabet/: Unexpected error in _crawl_web at line 579 in _crawl_web (../../local/lib/python3.13/site-packages/crawl4ai/async_crawler_strategy.py):
2025-03-18T16:42:05.311Z Error: Failed on navigating ACS-GOTO:
2025-03-18T16:42:05.313Z Page.goto: net::ERR_TUNNEL_CONNECTION_FAILED at https://apuestas.guru/sports/librabet/
2025-03-18T16:42:05.314Z Call log:
2025-03-18T16:42:05.316Z   - navigating to "https://apuestas.guru/sports/librabet/", waiting until "networkidle"
2025-03-18T16:42:05.317Z 
2025-03-18T16:42:05.318Z 
2025-03-18T16:42:05.319Z Code context:
2025-03-18T16:42:05.320Z  574                       response = await page.goto(
2025-03-18T16:42:05.321Z  575                           url, wait_until=config.wait_until, timeout=config.page_timeout
2025-03-18T16:42:05.322Z  576                       )
2025-03-18T16:42:05.323Z  577                       redirected_url = page.url
2025-03-18T16:42:05.324Z  578                   except Error as e:
2025-03-18T16:42:05.325Z  579 →                     raise RuntimeError(f"Failed on navigating ACS-GOTO:\n{str(e)}")
2025-03-18T16:42:05.326Z  580
2025-03-18T16:42:05.327Z  581                   await self.execute_hook(
2025-03-18T16:42:05.328Z  582                       "after_goto", page, context=context, url=url, response=response, config=config
2025-03-18T16:42:05.329Z  583                   )
2025-03-18T16:42:05.330Z  584
2025-03-18T16:42:05.559Z Attempt 3 failed for goproxies
2025-03-18T16:42:05.560Z Attempt 1/3 with brightdata for BR
2025-03-18T16:42:05.561Z Attempting crawl with brightdata proxy for https://apuestas.guru/sports/librabet/
2025-03-18T16:42:05.562Z Starting crawl for desktop version of https://apuestas.guru/sports/librabet/ in BR using brightdata proxy
2025-03-18T16:42:05.637Z Starting crawl for mobile version of https://apuestas.guru/sports/librabet/ in BR using brightdata proxy
2025-03-18T16:42:07.273Z [INIT].... → Crawl4AI 0.5.0.post4
2025-03-18T16:42:07.398Z [INIT].... → Crawl4AI 0.5.0.post4
2025-03-18T16:42:09.639Z [CONSOLE]. ℹ Console: Failed to load resource: the server responded with a status of 407 ()
2025-03-18T16:42:09.768Z [CONSOLE]. ℹ Console: Failed to load resource: the server responded with a status of 407 ()
2025-03-18T16:42:09.769Z [ERROR]... × https://apuestas.guru/sports/librabet/... | Error:
2025-03-18T16:42:09.770Z ┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
2025-03-18T16:42:09.772Z │ × Unexpected error in _crawl_web at line 579 in _crawl_web (../../local/lib/python3.13/site-                          │
2025-03-18T16:42:09.773Z │ packages/crawl4ai/async_crawler_strategy.py):                                                                         │
2025-03-18T16:42:09.774Z │   Error: Failed on navigating ACS-GOTO:                                                                               │
2025-03-18T16:42:09.775Z │   Page.goto: net::ERR_HTTP_RESPONSE_CODE_FAILURE at https://apuestas.guru/sports/librabet/                            │
2025-03-18T16:42:09.776Z │   Call log:                                                                                                           │
2025-03-18T16:42:09.777Z │   - navigating to "https://apuestas.guru/sports/librabet/", waiting until "networkidle"                               │
2025-03-18T16:42:09.778Z │                                                                                                                       │
2025-03-18T16:42:09.779Z │                                                                                                                       │
2025-03-18T16:42:09.780Z │   Code context:                                                                                                       │
2025-03-18T16:42:09.781Z │   574                       response = await page.goto(                                                               │
2025-03-18T16:42:09.782Z │   575                           url, wait_until=config.wait_until, timeout=config.page_timeout                        │
2025-03-18T16:42:09.784Z │   576                       )                                                                                         │
2025-03-18T16:42:09.785Z │   577                       redirected_url = page.url                                                                 │
2025-03-18T16:42:09.786Z │   578                   except Error as e:                                                                            │
2025-03-18T16:42:09.787Z │   579 →                     raise RuntimeError(f"Failed on navigating ACS-GOTO:\n{str(e)}")                           │
2025-03-18T16:42:09.788Z │   580                                                                                                                 │
2025-03-18T16:42:09.789Z │   581                   await self.execute_hook(                                                                      │
2025-03-18T16:42:09.792Z │   582                       "after_goto", page, context=context, url=url, response=response, config=config            │
2025-03-18T16:42:09.793Z │   583                   )                                                                                             │
2025-03-18T16:42:09.794Z │   584                                                                                                                 │
2025-03-18T16:42:09.796Z └───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
2025-03-18T16:42:09.797Z 
2025-03-18T16:42:09.798Z Failed to crawl https://apuestas.guru/sports/librabet/: Unexpected error in _crawl_web at line 579 in _crawl_web (../../local/lib/python3.13/site-packages/crawl4ai/async_crawler_strategy.py):
2025-03-18T16:42:09.799Z Error: Failed on navigating ACS-GOTO:
2025-03-18T16:42:09.800Z Page.goto: net::ERR_HTTP_RESPONSE_CODE_FAILURE at https://apuestas.guru/sports/librabet/
2025-03-18T16:42:09.802Z Call log:
2025-03-18T16:42:09.803Z   - navigating to "https://apuestas.guru/sports/librabet/", waiting until "networkidle"
2025-03-18T16:42:09.804Z 
2025-03-18T16:42:09.805Z 
2025-03-18T16:42:09.807Z Code context:
2025-03-18T16:42:09.808Z  574                       response = await page.goto(
2025-03-18T16:42:09.811Z  575                           url, wait_until=config.wait_until, timeout=config.page_timeout
2025-03-18T16:42:09.812Z  576                       )
2025-03-18T16:42:09.813Z  577                       redirected_url = page.url
2025-03-18T16:42:09.814Z  578                   except Error as e:
2025-03-18T16:42:09.815Z  579 →                     raise RuntimeError(f"Failed on navigating ACS-GOTO:\n{str(e)}")
2025-03-18T16:42:09.816Z  580
2025-03-18T16:42:09.817Z  581                   await self.execute_hook(
2025-03-18T16:42:09.818Z  582                       "after_goto", page, context=context, url=url, response=response, config=config
2025-03-18T16:42:09.819Z  583                   )
2025-03-18T16:42:09.820Z  584
2025-03-18T16:42:09.821Z [ERROR]... × https://apuestas.guru/sports/librabet/... | Error:
2025-03-18T16:42:09.822Z ┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
2025-03-18T16:42:09.823Z │ × Unexpected error in _crawl_web at line 579 in _crawl_web (../../local/lib/python3.13/site-                          │
2025-03-18T16:42:09.824Z │ packages/crawl4ai/async_crawler_strategy.py):                                                                         │
2025-03-18T16:42:09.825Z │   Error: Failed on navigating ACS-GOTO:                                                                               │
2025-03-18T16:42:09.826Z │   Page.goto: net::ERR_HTTP_RESPONSE_CODE_FAILURE at https://apuestas.guru/sports/librabet/                            │
2025-03-18T16:42:09.827Z │   Call log:                                                                                                           │
2025-03-18T16:42:09.828Z │   - navigating to "https://apuestas.guru/sports/librabet/", waiting until "networkidle"                               │
2025-03-18T16:42:09.830Z │                                                                                                                       │
2025-03-18T16:42:09.831Z │                                                                                                                       │
2025-03-18T16:42:09.832Z │   Code context:                                                                                                       │
2025-03-18T16:42:09.833Z │   574                       response = await page.goto(                                                               │
2025-03-18T16:42:09.834Z │   575                           url, wait_until=config.wait_until, timeout=config.page_timeout                        │
2025-03-18T16:42:09.835Z │   576                       )                                                                                         │
2025-03-18T16:42:09.836Z │   577                       redirected_url = page.url                                                                 │
2025-03-18T16:42:09.839Z │   578                   except Error as e:                                                                            │
2025-03-18T16:42:09.841Z │   579 →                     raise RuntimeError(f"Failed on navigating ACS-GOTO:\n{str(e)}")                           │
2025-03-18T16:42:09.842Z │   580                                                                                                                 │
2025-03-18T16:42:09.843Z │   581                   await self.execute_hook(                                                                      │
2025-03-18T16:42:09.845Z │   582                       "after_goto", page, context=context, url=url, response=response, config=config            │
2025-03-18T16:42:09.846Z │   583                   )                                                                                             │
2025-03-18T16:42:09.847Z │   584                                                                                                                 │
2025-03-18T16:42:09.848Z └───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
2025-03-18T16:42:09.854Z 
2025-03-18T16:42:09.857Z Failed to crawl https://apuestas.guru/sports/librabet/: Unexpected error in _crawl_web at line 579 in _crawl_web (../../local/lib/python3.13/site-packages/crawl4ai/async_crawler_strategy.py):
2025-03-18T16:42:09.859Z Error: Failed on navigating ACS-GOTO:
2025-03-18T16:42:09.860Z Page.goto: net::ERR_HTTP_RESPONSE_CODE_FAILURE at https://apuestas.guru/sports/librabet/
2025-03-18T16:42:09.861Z Call log:
2025-03-18T16:42:09.862Z   - navigating to "https://apuestas.guru/sports/librabet/", waiting until "networkidle"
2025-03-18T16:42:09.863Z 
2025-03-18T16:42:09.864Z 
2025-03-18T16:42:09.865Z Code context:
2025-03-18T16:42:09.866Z  574                       response = await page.goto(
2025-03-18T16:42:09.867Z  575                           url, wait_until=config.wait_until, timeout=config.page_timeout
2025-03-18T16:42:09.869Z  576                       )
2025-03-18T16:42:09.870Z  577                       redirected_url = page.url
2025-03-18T16:42:09.871Z  578                   except Error as e:
2025-03-18T16:42:09.872Z  579 →                     raise RuntimeError(f"Failed on navigating ACS-GOTO:\n{str(e)}")
2025-03-18T16:42:09.873Z  580
2025-03-18T16:42:09.874Z  581                   await self.execute_hook(
2025-03-18T16:42:09.875Z  582                       "after_goto", page, context=context, url=url, response=response, config=config
2025-03-18T16:42:09.876Z  583                   )
2025-03-18T16:42:09.877Z  584
2025-03-18T16:42:10.280Z Attempt 1 failed for brightdata
2025-03-18T16:42:10.281Z Attempt 2/3 with brightdata for BR
2025-03-18T16:42:10.282Z Attempting crawl with brightdata proxy for https://apuestas.guru/sports/librabet/
2025-03-18T16:42:10.283Z Starting crawl for desktop version of https://apuestas.guru/sports/librabet/ in BR using brightdata proxy
2025-03-18T16:42:10.390Z Starting crawl for mobile version of https://apuestas.guru/sports/librabet/ in BR using brightdata proxy
2025-03-18T16:42:12.072Z [INIT].... → Crawl4AI 0.5.0.post4
2025-03-18T16:42:12.184Z [INIT].... → Crawl4AI 0.5.0.post4
2025-03-18T16:42:14.374Z [CONSOLE]. ℹ Console: Failed to load resource: the server responded with a status of 407 ()
2025-03-18T16:42:14.572Z [ERROR]... × https://apuestas.guru/sports/librabet/... | Error:
2025-03-18T16:42:14.573Z ┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
2025-03-18T16:42:14.574Z │ × Unexpected error in _crawl_web at line 579 in _crawl_web (../../local/lib/python3.13/site-                          │
2025-03-18T16:42:14.575Z │ packages/crawl4ai/async_crawler_strategy.py):                                                                         │
2025-03-18T16:42:14.576Z │   Error: Failed on navigating ACS-GOTO:                                                                               │
2025-03-18T16:42:14.577Z │   Page.goto: net::ERR_HTTP_RESPONSE_CODE_FAILURE at https://apuestas.guru/sports/librabet/                            │
2025-03-18T16:42:14.578Z │   Call log:                                                                                                           │
2025-03-18T16:42:14.579Z │   - navigating to "https://apuestas.guru/sports/librabet/", waiting until "networkidle"                               │
2025-03-18T16:42:14.580Z │                                                                                                                       │
2025-03-18T16:42:14.581Z │                                                                                                                       │
2025-03-18T16:42:14.582Z │   Code context:                                                                                                       │
2025-03-18T16:42:14.583Z │   574                       response = await page.goto(                                                               │
2025-03-18T16:42:14.584Z │   575                           url, wait_until=config.wait_until, timeout=config.page_timeout                        │
2025-03-18T16:42:14.585Z │   576                       )                                                                                         │
2025-03-18T16:42:14.586Z │   577                       redirected_url = page.url                                                                 │
2025-03-18T16:42:14.587Z │   578                   except Error as e:                                                                            │
2025-03-18T16:42:14.588Z │   579 →                     raise RuntimeError(f"Failed on navigating ACS-GOTO:\n{str(e)}")                           │
2025-03-18T16:42:14.589Z │   580                                                                                                                 │
2025-03-18T16:42:14.590Z │   581                   await self.execute_hook(                                                                      │
2025-03-18T16:42:14.591Z │   582                       "after_goto", page, context=context, url=url, response=response, config=config            │
2025-03-18T16:42:14.592Z │   583                   )                                                                                             │
2025-03-18T16:42:14.593Z │   584                                                                                                                 │
2025-03-18T16:42:14.600Z └───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
2025-03-18T16:42:14.602Z 
2025-03-18T16:42:14.602Z Failed to crawl https://apuestas.guru/sports/librabet/: Unexpected error in _crawl_web at line 579 in _crawl_web (../../local/lib/python3.13/site-packages/crawl4ai/async_crawler_strategy.py):
2025-03-18T16:42:14.603Z Error: Failed on navigating ACS-GOTO:
2025-03-18T16:42:14.604Z Page.goto: net::ERR_HTTP_RESPONSE_CODE_FAILURE at https://apuestas.guru/sports/librabet/
2025-03-18T16:42:14.605Z Call log:
2025-03-18T16:42:14.606Z   - navigating to "https://apuestas.guru/sports/librabet/", waiting until "networkidle"
2025-03-18T16:42:14.607Z 
2025-03-18T16:42:14.608Z 
2025-03-18T16:42:14.609Z Code context:
2025-03-18T16:42:14.610Z  574                       response = await page.goto(
2025-03-18T16:42:14.611Z  575                           url, wait_until=config.wait_until, timeout=config.page_timeout
2025-03-18T16:42:14.612Z  576                       )
2025-03-18T16:42:14.613Z  577                       redirected_url = page.url
2025-03-18T16:42:14.614Z  578                   except Error as e:
2025-03-18T16:42:14.615Z  579 →                     raise RuntimeError(f"Failed on navigating ACS-GOTO:\n{str(e)}")
2025-03-18T16:42:14.616Z  580
2025-03-18T16:42:14.617Z  581                   await self.execute_hook(
2025-03-18T16:42:14.618Z  582                       "after_goto", page, context=context, url=url, response=response, config=config
2025-03-18T16:42:14.619Z  583                   )
2025-03-18T16:42:14.620Z  584
2025-03-18T16:42:14.666Z [CONSOLE]. ℹ Console: Failed to load resource: the server responded with a status of 407 ()
2025-03-18T16:42:14.775Z [ERROR]... × https://apuestas.guru/sports/librabet/... | Error:
2025-03-18T16:42:14.777Z ┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
2025-03-18T16:42:14.778Z │ × Unexpected error in _crawl_web at line 579 in _crawl_web (../../local/lib/python3.13/site-                          │
2025-03-18T16:42:14.779Z │ packages/crawl4ai/async_crawler_strategy.py):                                                                         │
2025-03-18T16:42:14.780Z │   Error: Failed on navigating ACS-GOTO:                                                                               │
2025-03-18T16:42:14.781Z │   Page.goto: net::ERR_HTTP_RESPONSE_CODE_FAILURE at https://apuestas.guru/sports/librabet/                            │
2025-03-18T16:42:14.782Z │   Call log:                                                                                                           │
2025-03-18T16:42:14.783Z │   - navigating to "https://apuestas.guru/sports/librabet/", waiting until "networkidle"                               │
2025-03-18T16:42:14.784Z │                                                                                                                       │
2025-03-18T16:42:14.784Z │                                                                                                                       │
2025-03-18T16:42:14.785Z │   Code context:                                                                                                       │
2025-03-18T16:42:14.786Z │   574                       response = await page.goto(                                                               │
2025-03-18T16:42:14.787Z │   575                           url, wait_until=config.wait_until, timeout=config.page_timeout                        │
2025-03-18T16:42:14.788Z │   576                       )                                                                                         │
2025-03-18T16:42:14.789Z │   577                       redirected_url = page.url                                                                 │
2025-03-18T16:42:14.790Z │   578                   except Error as e:                                                                            │
2025-03-18T16:42:14.791Z │   579 →                     raise RuntimeError(f"Failed on navigating ACS-GOTO:\n{str(e)}")                           │
2025-03-18T16:42:14.792Z │   580                                                                                                                 │
2025-03-18T16:42:14.793Z │   581                   await self.execute_hook(                                                                      │
2025-03-18T16:42:14.794Z │   582                       "after_goto", page, context=context, url=url, response=response, config=config            │
2025-03-18T16:42:14.795Z │   583                   )                                                                                             │
2025-03-18T16:42:14.796Z │   584                                                                                                                 │
2025-03-18T16:42:14.797Z └───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
2025-03-18T16:42:14.798Z 
2025-03-18T16:42:14.799Z Failed to crawl https://apuestas.guru/sports/librabet/: Unexpected error in _crawl_web at line 579 in _crawl_web (../../local/lib/python3.13/site-packages/crawl4ai/async_crawler_strategy.py):
2025-03-18T16:42:14.800Z Error: Failed on navigating ACS-GOTO:
2025-03-18T16:42:14.800Z Page.goto: net::ERR_HTTP_RESPONSE_CODE_FAILURE at https://apuestas.guru/sports/librabet/
2025-03-18T16:42:14.802Z Call log:
2025-03-18T16:42:14.802Z   - navigating to "https://apuestas.guru/sports/librabet/", waiting until "networkidle"
2025-03-18T16:42:14.804Z 
2025-03-18T16:42:14.805Z 
2025-03-18T16:42:14.807Z Code context:
2025-03-18T16:42:14.807Z  574                       response = await page.goto(
2025-03-18T16:42:14.808Z  575                           url, wait_until=config.wait_until, timeout=config.page_timeout
2025-03-18T16:42:14.810Z  576                       )
2025-03-18T16:42:14.811Z  577                       redirected_url = page.url
2025-03-18T16:42:14.812Z  578                   except Error as e:
2025-03-18T16:42:14.813Z  579 →                     raise RuntimeError(f"Failed on navigating ACS-GOTO:\n{str(e)}")
2025-03-18T16:42:14.814Z  580
2025-03-18T16:42:14.815Z  581                   await self.execute_hook(
2025-03-18T16:42:14.816Z  582                       "after_goto", page, context=context, url=url, response=response, config=config
2025-03-18T16:42:14.817Z  583                   )
2025-03-18T16:42:14.818Z  584
2025-03-18T16:42:15.186Z Attempt 2 failed for brightdata
2025-03-18T16:42:15.187Z Attempt 3/3 with brightdata for BR
2025-03-18T16:42:15.188Z Attempting crawl with brightdata proxy for https://apuestas.guru/sports/librabet/
2025-03-18T16:42:15.189Z Starting crawl for desktop version of https://apuestas.guru/sports/librabet/ in BR using brightdata proxy
2025-03-18T16:42:15.287Z Starting crawl for mobile version of https://apuestas.guru/sports/librabet/ in BR using brightdata proxy
2025-03-18T16:42:16.873Z [INIT].... → Crawl4AI 0.5.0.post4
2025-03-18T16:42:17.194Z [INIT].... → Crawl4AI 0.5.0.post4
2025-03-18T16:42:19.072Z [CONSOLE]. ℹ Console: Failed to load resource: the server responded with a status of 407 ()
2025-03-18T16:42:19.172Z [ERROR]... × https://apuestas.guru/sports/librabet/... | Error:
2025-03-18T16:42:19.175Z ┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
2025-03-18T16:42:19.176Z │ × Unexpected error in _crawl_web at line 579 in _crawl_web (../../local/lib/python3.13/site-                          │
2025-03-18T16:42:19.177Z │ packages/crawl4ai/async_crawler_strategy.py):                                                                         │
2025-03-18T16:42:19.178Z │   Error: Failed on navigating ACS-GOTO:                                                                               │
2025-03-18T16:42:19.179Z │   Page.goto: net::ERR_HTTP_RESPONSE_CODE_FAILURE at https://apuestas.guru/sports/librabet/                            │
2025-03-18T16:42:19.181Z │   Call log:                                                                                                           │
2025-03-18T16:42:19.183Z │   - navigating to "https://apuestas.guru/sports/librabet/", waiting until "networkidle"                               │
2025-03-18T16:42:19.184Z │                                                                                                                       │
2025-03-18T16:42:19.185Z │                                                                                                                       │
2025-03-18T16:42:19.186Z │   Code context:                                                                                                       │
2025-03-18T16:42:19.187Z │   574                       response = await page.goto(                                                               │
2025-03-18T16:42:19.188Z │   575                           url, wait_until=config.wait_until, timeout=config.page_timeout                        │
2025-03-18T16:42:19.188Z │   576                       )                                                                                         │
2025-03-18T16:42:19.190Z │   577                       redirected_url = page.url                                                                 │
2025-03-18T16:42:19.190Z │   578                   except Error as e:                                                                            │
2025-03-18T16:42:19.191Z │   579 →                     raise RuntimeError(f"Failed on navigating ACS-GOTO:\n{str(e)}")                           │
2025-03-18T16:42:19.192Z │   580                                                                                                                 │
2025-03-18T16:42:19.193Z │   581                   await self.execute_hook(                                                                      │
2025-03-18T16:42:19.194Z │   582                       "after_goto", page, context=context, url=url, response=response, config=config            │
2025-03-18T16:42:19.195Z │   583                   )                                                                                             │
2025-03-18T16:42:19.196Z │   584                                                                                                                 │
2025-03-18T16:42:19.197Z └───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
2025-03-18T16:42:19.198Z 
2025-03-18T16:42:19.199Z Failed to crawl https://apuestas.guru/sports/librabet/: Unexpected error in _crawl_web at line 579 in _crawl_web (../../local/lib/python3.13/site-packages/crawl4ai/async_crawler_strategy.py):
2025-03-18T16:42:19.200Z Error: Failed on navigating ACS-GOTO:
2025-03-18T16:42:19.201Z Page.goto: net::ERR_HTTP_RESPONSE_CODE_FAILURE at https://apuestas.guru/sports/librabet/
2025-03-18T16:42:19.202Z Call log:
2025-03-18T16:42:19.203Z   - navigating to "https://apuestas.guru/sports/librabet/", waiting until "networkidle"
2025-03-18T16:42:19.204Z 
2025-03-18T16:42:19.205Z 
2025-03-18T16:42:19.208Z Code context:
2025-03-18T16:42:19.209Z  574                       response = await page.goto(
2025-03-18T16:42:19.210Z  575                           url, wait_until=config.wait_until, timeout=config.page_timeout
2025-03-18T16:42:19.211Z  576                       )
2025-03-18T16:42:19.212Z  577                       redirected_url = page.url
2025-03-18T16:42:19.213Z  578                   except Error as e:
2025-03-18T16:42:19.214Z  579 →                     raise RuntimeError(f"Failed on navigating ACS-GOTO:\n{str(e)}")
2025-03-18T16:42:19.215Z  580
2025-03-18T16:42:19.216Z  581                   await self.execute_hook(
2025-03-18T16:42:19.217Z  582                       "after_goto", page, context=context, url=url, response=response, config=config
2025-03-18T16:42:19.218Z  583                   )
2025-03-18T16:42:19.219Z  584
2025-03-18T16:42:19.378Z [CONSOLE]. ℹ Console: Failed to load resource: the server responded with a status of 407 ()
2025-03-18T16:42:19.479Z [ERROR]... × https://apuestas.guru/sports/librabet/... | Error:
2025-03-18T16:42:19.480Z ┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
2025-03-18T16:42:19.481Z │ × Unexpected error in _crawl_web at line 579 in _crawl_web (../../local/lib/python3.13/site-                          │
2025-03-18T16:42:19.482Z │ packages/crawl4ai/async_crawler_strategy.py):                                                                         │
2025-03-18T16:42:19.483Z │   Error: Failed on navigating ACS-GOTO:                                                                               │
2025-03-18T16:42:19.484Z │   Page.goto: net::ERR_HTTP_RESPONSE_CODE_FAILURE at https://apuestas.guru/sports/librabet/                            │
2025-03-18T16:42:19.493Z │   Call log:                                                                                                           │
2025-03-18T16:42:19.498Z │   - navigating to "https://apuestas.guru/sports/librabet/", waiting until "networkidle"                               │
2025-03-18T16:42:19.499Z │                                                                                                                       │
2025-03-18T16:42:19.500Z │                                                                                                                       │
2025-03-18T16:42:19.501Z │   Code context:                                                                                                       │
2025-03-18T16:42:19.502Z │   574                       response = await page.goto(                                                               │
2025-03-18T16:42:19.503Z │   575                           url, wait_until=config.wait_until, timeout=config.page_timeout                        │
2025-03-18T16:42:19.505Z │   576                       )                                                                                         │
2025-03-18T16:42:19.506Z │   577                       redirected_url = page.url                                                                 │
2025-03-18T16:42:19.507Z │   578                   except Error as e:                                                                            │
2025-03-18T16:42:19.509Z │   579 →                     raise RuntimeError(f"Failed on navigating ACS-GOTO:\n{str(e)}")                           │
2025-03-18T16:42:19.510Z │   580                                                                                                                 │
2025-03-18T16:42:19.512Z │   581                   await self.execute_hook(                                                                      │
2025-03-18T16:42:19.513Z │   582                       "after_goto", page, context=context, url=url, response=response, config=config            │
2025-03-18T16:42:19.517Z │   583                   )                                                                                             │
2025-03-18T16:42:19.517Z │   584                                                                                                                 │
2025-03-18T16:42:19.519Z └───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
2025-03-18T16:42:19.520Z 
2025-03-18T16:42:19.521Z Failed to crawl https://apuestas.guru/sports/librabet/: Unexpected error in _crawl_web at line 579 in _crawl_web (../../local/lib/python3.13/site-packages/crawl4ai/async_crawler_strategy.py):
2025-03-18T16:42:19.521Z Error: Failed on navigating ACS-GOTO:
2025-03-18T16:42:19.522Z Page.goto: net::ERR_HTTP_RESPONSE_CODE_FAILURE at https://apuestas.guru/sports/librabet/
2025-03-18T16:42:19.524Z Call log:
2025-03-18T16:42:19.525Z   - navigating to "https://apuestas.guru/sports/librabet/", waiting until "networkidle"
2025-03-18T16:42:19.526Z 
2025-03-18T16:42:19.527Z 
2025-03-18T16:42:19.527Z Code context:
2025-03-18T16:42:19.528Z  574                       response = await page.goto(
2025-03-18T16:42:19.529Z  575                           url, wait_until=config.wait_until, timeout=config.page_timeout
2025-03-18T16:42:19.530Z  576                       )
2025-03-18T16:42:19.533Z  577                       redirected_url = page.url
2025-03-18T16:42:19.534Z  578                   except Error as e:
2025-03-18T16:42:19.535Z  579 →                     raise RuntimeError(f"Failed on navigating ACS-GOTO:\n{str(e)}")
2025-03-18T16:42:19.536Z  580
2025-03-18T16:42:19.536Z  581                   await self.execute_hook(
2025-03-18T16:42:19.537Z  582                       "after_goto", page, context=context, url=url, response=response, config=config
2025-03-18T16:42:19.538Z  583                   )
2025-03-18T16:42:19.541Z  584
2025-03-18T16:42:19.838Z Attempt 3 failed for brightdata
2025-03-18T16:42:19.840Z All attempts failed for BR
2025-03-18T16:42:19.841Z [apify] INFO  Exiting Actor ({"exit_code": 1})
2025-03-18T16:42:20.344Z 

I’m currently using Crawl4AI with Apify to crawl web pages, but I’m encountering a proxy authentication error (HTTP 407) when trying to access webpages using proxy. I've checked my proxy settings, but I'm not sure what else to try.
Additionally, after running the crawl, I’m not getting any HTML or screenshot results.
Could anyone provide advice on how to resolve these issues? Any help would be greatly appreciated!
Thank you!

[aravindkarnam]

@prokhorenkomykhailo Quite a lot to unpack here 😁.

Both from the error codes and based on the logs, this doesn't appear to be a crawl4AI or proxy issue. I checked one of the links here. For eg:https://apuestas.guru/sports/librabet/

It has a self signed certificate and the browser was immediately blocking it. When I tried with http:// protocol instead, the firewall at my office blocked it. So this page seems to create genuine trust issues, spooking the network to block it (I mean even your proxy provider will have some kind of protection on their infra).

[prokhorenkomykhailo]

Hi @aravindkarnam ,
Thank you for your earlier explanation about the self-signed certificate and HTTP issues with the website I’m trying to crawl. I understand that the website is causing trust issues due to its self-signed certificate and insecure HTTP protocol, which is why it’s being blocked by browsers, firewalls, and proxies.

Could you please guide me on what exactly needs to be done to fix this issue? Specifically:

If I need to bypass the SSL verification for the self-signed certificate, how can I configure my crawler (Crawl4AI + Playwright) to do this safely?

If the website owner needs to fix the SSL certificate, what steps should they take to get a valid SSL certificate and enable HTTPS?

Are there any alternative solutions or workarounds I can use to successfully crawl this website without compromising security?

I’d really appreciate your help in resolving this. Thank you in advance!