How to verify source content? (to eliminate the scenario of being throttled)

[cthulhubuddha]

I am successfully pulling the content of a source page using SmartScraperGraph and issuing prompts to a local LLM. However, the results of my prompts to the local LLM indicate the content may be being throttled during the scrape. For example, in a browser, the source page has 200 listings, but using SmartScraperGraph with multiple different prompts to the LLM show the LLM only sees 10.

How can I dump the entire contents of what was scraped to a file so I can compare to the page's source code?

[dosubot]

Hi @cthulhubuddha! I'm Dosu and I’m helping the Scrapegraph-ai team.

To verify the full source content and check for throttling, the best approach is to fetch the raw HTML yourself and save it to a file before passing it to SmartScraperGraph. You can use Python tools like requests, Selenium, or the built-in scrape_do_fetch function. For example, with scrape_do_fetch:

from scrape_do import scrape_do_fetch

token = 'your_api_token'
target_url = 'http://example.com'
use_proxy = False
geoCode = None
super_proxy = False

raw_html = scrape_do_fetch(token, target_url, use_proxy, geoCode, super_proxy)

with open('webpage.html', 'w', encoding='utf-8') as file:
    file.write(raw_html)

(source)

You can then compare webpage.html to the browser's source code. To ensure SmartScraperGraph sees the full content, read the HTML file and pass its contents as the source parameter:

with open('webpage.html', 'r', encoding='utf-8') as file:
    html_content = file.read()

smart_scraper_graph = SmartScraperGraph(
    prompt="your prompt here",
    source=html_content,
    config=graph_config
)
result = smart_scraper_graph.run()

(source)

This lets you verify exactly what was scraped and sent to the LLM, helping you spot any throttling or missing data.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[cthulhubuddha]

I have scrapegraphai installed in my venv via pip, and can successfully import "from scrapegraphai.graphs import SmartScraperGraph", but your suggestion to import "from scrape_do import scrape_do_fetch" fails. I see scrape_do is located in extras in this project repo but cannot import it. Please illustrate how to install scrape_do so I can "from scrape_do import scrape_do_fetch".