Question regarding correct json body for actual results: Cant seem to get any meaningful and reliable results!

[Valker03]

The only thing I dont seem to get to work is a proper scrape of the website and its information. Do you know any videos or guidance on how to properly get information from a side.

I tried a deepcrawl strategy but its just not getting it and it takes millions of input tokens just for basically no output whatsoever... the ai model is then prompted with the entire website basically and should extract information from that which obviously does not work to the precision that we would like to achieve.

{ "urls": [ "https://www.kinross.com/" ], "browser_config": { "type": "BrowserConfig", "params": { "headless": true } }, "crawler_config": { "type": "CrawlerRunConfig", "params": { "deep_crawl_strategy": { "type": "BFSDeepCrawlStrategy", "params": { "max_depth": 2, "max_pages": 20, "include_external": false } }, "extraction_strategy": { "type": "LLMExtractionStrategy", "params": { "llm_config": { "type": "LLMConfig", "params": { "provider": "gemini/gemini-2.5-flash", "api_token": "env:GOOGLE_API_KEY" } }, "instruction": "Extract mine names and locations. Return JSON only.", "schema": { "type": "dict", "value": { "type": "object", "properties": { "mines": { "type": "array", "items": { "type": "object", "properties": { "name": { "type": "string" }, "location": { "type": "string" } } } } } } } } } } } }

[jverneuer]

Let's make your config easier for the next person to read:

     "llm_config":{
        "type":"LLMConfig",
        "params":{
           "provider":"gemini/gemini-2.5-flash",
           "api_token":"env:GOOGLE_API_KEY"
        }},
     "instruction":"Extract mine names and locations. Return JSON only.",
     "schema":{
       "type":"dict",
        "value":{
           "type":"object",
           "properties":{
              "mines":{
                 "type":"array",
                 "items":{
                    "type":"object",
                    "properties":{
                       "name":{
                          "type":"string"
                       },
                       "location":{
                          "type":"string"
    }}}}}}}

From looking at the site, extracting the mine data directly with an LLM will be difficult.
The content is mostly rendered after user interaction, the mine info isn't clearly visible to the model.

A more reliable approach would be:

• Inline JavaScript extraction (scrape values defined in scripts).
• CSS selector strategy (cheap, simple, effective).

On the operations/projects pages, the data is basically ready for crawling.
The CSS selector [data-tag] a already gives you mine names and locations --- just split the resulting string at ",".

You can also inject JavaScript in your browser to test:

    let result = [];
    document.querySelectorAll('[data-tag] a').forEach((v,k)=>{
       result.push({
            name: v.innerText.split(",")[0],
            location: v.innerText.split(",")[1]
       })
    })

    console.log(result);

Replace the console.log with whatever your crawler needs, and you're set.

In this case, an LLM is probably overkill. Still, if you want to push it,
you could try a carefully phrased prompt that makes the model digharder.
That would most likely yield better results on other websites too.
Something like:

    You are an information extraction assistant.
    You are given the full crawled content of a company’s mine information webpage.

    Your task: Identify all mines and their locations, even if the information is 
    not explicitly written in the visible text.

    Follow these steps carefully:
    - Extract mines and locations from visible text
    - Look at headings, paragraphs, tables, and bullet points.
    
    Search hidden sources
    - Inspect file names of downloadable resources (e.g., “gold-mine-kalgoorlie.pdf” → Mine: Kalgoorlie Gold Mine, Location: Kalgoorlie).
    - Look at URLs, alt-text, image names, or metadata that might reveal names/locations.
    - Inspect JSON/JS variables that define locations or mine IDs.
    - Consider abbreviations or codes that correspond to locations (e.g., “WA” = Western Australia).
    
    Cross-check consistency
    - If multiple sources point to the same mine/location, consolidate them.
    - If a location is implied (e.g., filename: “iron_ore_pilbara_map.pdf”), infer the mine is in Pilbara.
   
    Output in structured JSON:    
    {your schema here}

Important for this particular page:

• Sending markdown to the LLM strips the data.
• You must send the raw HTML to make it work.

inject js with c4a

javascript execution

css extraction

[Valker03]

Thanks. I also realized that Kinrossgold in my case (I tried it with Firecrawl as well) seems to be the worst/toughest example that I could have chosen for the start. But anyways I havent found a way to properly extract the data from that site since they use "#" references so every mine doesnt have their own page...
But as I would like to build a solution that works for all of them I would like to build something that not only works for Kinrossgold where I already looked at the website structure but obviously for all mining companies so in this case I need to build something robust. Any idea @jverneuer ?

[jverneuer]

Hey @Valker03 , I think you're already on the right track with tools like firecrawl or crawl4ai.
But there's no magic one-click solution that will fix crawling issues forever---unless you throw some humans at it and have them updating spreadsheets 😄.

The challenge with crawling is that things inevitably break and need fixing.
-- every day some developer or designer wakes up and breaks things. --
The real question is: how valuable is this information to you?

What I'd do is evaluate the available solutions and the total cost of getting that information.
If you want a generic solution with high accuracy and minimal ongoing fixes, you'll either need humans or an good LLM based system.
And honestly, humans will outperform LLMs---our brains are still the best pattern matchers around.

A big factor is breakage. So ask yourself: what's the cost difference between having50%, 80% 90% .... 100% accuracy?
And how time-critical is the data?
From your situation, I'd guess either time sensitivity is high, or "someone updating a spreadsheet once a week" didn't work out. But of course, you know your problem best.

Here's a practical test you can run:

1. Take some of your target pages and paste the HTML into an LLM.
2. (Optional) Convert the site to Markdown first using an online tool
3. prompt the LLM to generate CSS or xpath selectors for extraction
4. Test those selectors directly in the browser on the target site (DevTools → Elements → Cmd+F/Ctrl+F).

• If the selectors match the right content: great!
• If not → try a different prompt or move on to the next website

Do this across a decent sample of your sites and you'll quickly see how far an LLM-based extraction strategy can take you.
If it works, you can build a lightweight crawler that uses the selectors most of the time, and only falls back to the LLM when nothing is found.
There's actually an example of this approach in the crawl4ai repo, though I can't find the exact link right now.

Another approach would be building a pipeline that doesn't just extract the fly, but uses LLM agents to refine the process and fix breakages dynamically.
That way, you'd have a knowledge extraction pipeline that adapts continuously---though it won't be error-free. That would be not really expensive to run but quite some work to build properly.

And if you happen to have a bag of cash under your desk looking for a new owner, ping me---I can build that pipeline for you. Otherwise, you'll find plenty of people willing to take that project on 😄.

If you want to go down this route yourself read everything in about agent types and agent orchestration in the google ADK docs that will give you the the basic concepts to start learning more about building such a system.

Your problem is harder than it seems but solvable. I found the initial solution to your problem while testing one of my extraction pipelines with your problem.