How to extract XPaths for all elements on a page using Crawl4AI?

[Dronanaik]

Hi Community,

I’ve been exploring Crawl4AI for web data extraction and wanted to clarify if it’s possible to automatically extract the XPaths of all elements present on a given screen/page.

Specifically, I’m looking for:

A way to capture the complete set of XPaths (not just selected elements).

Any built-in functionality in Crawl4AI that supports this.

If not directly available, suggested workarounds or integrations (e.g., with Selenium, Playwright, or BeautifulSoup).

Has anyone in the community tried this or implemented a similar feature?
Any guidance, examples, or best practices would be highly appreciated.

Thanks in advance!

[ntohidi]

@Dronanaik
Hi, while Crawl4AI doesn't have a built-in "extract all XPaths" function specifically, you can definitely achieve this with a simple approach using the browser automation capabilities. like Crawl4AI makes it easy to execute JavaScript on a page, which is perfect for this use case:

async def extract_all_xpaths(url):
    # JavaScript that finds all elements and computes their XPaths
    js_code = """
    function getXPath(element) {
        if (element.id !== '') 
            return '//*[@id="' + element.id + '"]';
        
        if (element === document.body) 
            return '/html/body';
        
        let ix = 0;
        const siblings = element.parentNode.childNodes;
        
        for (let i = 0; i < siblings.length; i++) {
            const sibling = siblings[i];
            if (sibling === element)
                return getXPath(element.parentNode) + '/' + element.tagName.toLowerCase() + '[' + (ix + 1) + ']';
            if (sibling.nodeType === 1 && sibling.tagName === element.tagName)
                ix++;
        }
    }
    
    // Collect all elements
    const allElements = document.querySelectorAll('*');
    const result = [];
    
    // For each element, compute XPath and basic info
    allElements.forEach(el => {
        // Skip script and style elements
        if (el.tagName.toLowerCase() === 'script' || el.tagName.toLowerCase() === 'style')
            return;
            
        // Compute attributes
        const attributes = {};
        for (const attr of el.attributes) {
            attributes[attr.name] = attr.value;
        }
        
        result.push({
            xpath: getXPath(el),
            tag: el.tagName.toLowerCase(),
            text: el.innerText.substring(0, 50) + (el.innerText.length > 50 ? '...' : ''),
            attributes: attributes
        });
    });
    
    return result;
    """
    
    config = CrawlerRunConfig(
        js_code=js_code,
    )
    
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url, config=config)

        if result.success:
            # The JS execution result contains all elements and their XPaths
            elements = result.js_execution_result['results'][0]
            print(f"Extracted XPaths for {len(elements)} elements")
            
            # Example: Print first 5 XPaths with their tags
            for i, element in enumerate(elements[:5]):
                print(f"{i+1}. {element['xpath']} - <{element['tag']}> - {element['text']}")
                
            # Save full results to file
            with open('xpaths.json', 'w') as f:
                json.dump(elements, f, indent=2)
            print(f"All XPaths saved to xpaths.json")
            
            return elements
        else:
            print(f"Error: {result.error_message}")
            return []

[Dronanaik]

@ntohidi
Hi I am asking for the relative xpaths and also it is not working the code you have shared here,
it is saying crawl4ai does not have an attribute named js_execution_result.