Discrepancy between CLI behavior (v0.6.3) and GitHub documentation for crwl command

[Sheld0n-Cooper]

Dear Crawl4AI Developer,

I've encountered some discrepancies between the crwl command-line interface (CLI) behavior of the installed Crawl4AI version and the examples provided in your main GitHub repository documentation.

Context:

• Installed Crawl4AI Version: 0.6.3 (verified via pip show crawl4ai)
• Environment: Ubuntu WSL

Observed Discrepancies:

1. CLI Options (--deep-crawl, --max-pages vs. --crawler):

   • Your GitHub documentation (e.g., crwl https://docs.crawl4ai.com --deep-crawl bfs
     --max-pages 10) suggests the use of --deep-crawl and --max-pages for configuring
     the crawler.
   • However, when attempting to execute commands with this syntax using the crwl
     executable from the venv (e.g., venv/bin/crwl crawl ...), the command
     consistently returns the error: Error: No such option: --deep-crawl Did you mean
     --crawler?.
   • Furthermore, the output of venv/bin/crwl --help and venv/bin/crwl crawl --help
     does not list --deep-crawl or --max-pages. Instead, it indicates that crawler
     parameters should be passed via -c, --crawler TEXT (e.g., -c
     strategy=bfs,max_pages=X). This latter syntax is what I found to be functional
     for version 0.6.3.

2. Required Subcommand (crawl):

   • Some examples in your GitHub documentation (e.g., crwl
     https://www.nbcnews.com/business -o markdown) show crwl being used directly with
     the URL and options, without a subcommand.
   • In my 0.6.3 installation, the crwl executable consistently requires the crawl
     subcommand (i.e., crwl crawl [URL] ...).

Questions for Clarification:

1. Could you please clarify why the CLI behavior of the 0.6.3 version installed via pip
   differs from the command examples provided in the main GitHub documentation?

   • Does the documentation refer to a future release or a specific development branch?
   • Is there a particular configuration or installation step required to enable the
     --deep-crawl and --max-pages options in version 0.6.3?

2. For building interactive crawling workflows, it is crucial to programmatically obtain
   a structured list of all discovered URLs during a crawl (not just by parsing the
   generated Markdown output). Is there a recommended way or a specific crwl option to
   get this list directly in a machine-readable format (e.g., JSON)?

Impact:
These discrepancies lead to confusion for users attempting to follow the documented
examples and require trial-and-error to determine the correct command syntax for a given
installed version.

Suggestion:
Updating the GitHub documentation to accurately reflect the CLI behavior of the
currently released pip versions would be highly beneficial for users.

Thank you for your time and assistance.