Very large output for most `lookup_item`

[stargazing-dino]

Doing as the readme suggests:

cargo run --bin cratedocs test --tool lookup_item --crate-name tokio --item-path sync::mpsc::Sender

results in 1300 lines of text, 19037 tokens according to this: https://token-counter.app/anthropic/claude-3.7-sonnet

There's arguably a lot that can be removed just from having a quick look:

// ...

</details><details class="toggle method-toggle" open=""><summary>

[Source](https://docs.rs/tracing/0.1.41/x86_64-unknown-linux-gnu/src/tracing/instrument.rs.html#228)[§](#method.with_current_subscriber)

#### fn [with\_current\_subscriber](https://docs.rs/tracing/0.1.41/x86_64-unknown-linux-gnu/tracing/instrument/trait.WithSubscriber.html#method.with_current_subscriber)(self) -\> [WithDispatch](https://docs.rs/tracing/0.1.41/x86_64-unknown-linux-gnu/tracing/instrument/struct.WithDispatch.html)\<Self\> [ⓘ](#) ####

</summary>

Attaches the current [default](https://docs.rs/tracing/0.1.41/x86_64-unknown-linux-gnu/tracing/dispatcher/index.html#setting-the-default-subscriber) [`Subscriber`](https://docs.rs/tracing-core/0.1.33/x86_64-unknown-linux-gnu/tracing_core/subscriber/trait.Subscriber.html) to this type, returning a[`WithDispatch`](https://docs.rs/tracing/0.1.41/x86_64-unknown-linux-gnu/tracing/instrument/struct.WithDispatch.html) wrapper. [Read more](https://docs.rs/tracing/0.1.41/x86_64-unknown-linux-gnu/tracing/instrument/trait.WithSubscriber.html#method.with_current_subscriber)

</details>

</details>

{"Instrumented\<Self\>":"\<h3\>Notable traits for \<code\>\<a class=\\"struct\\" href=\\"https://docs.rs/tracing/0.1.41/x86\_64-unknown-linux-gnu/tracing/instrument/struct.Instrumented.html\\" title=\\"struct tracing::instrument::Instrumented\\"\>Instrumented\</a\>&lt;T&gt;\</code\>\</h3\>\<pre\>\<code\>\<div class=\\"where\\"\>impl&lt;T&gt; \<a class=\\"trait\\" href=\\"https://doc.rust-lang.org/nightly/core/future/future/trait.Future.html\\" title=\\"trait core::future::future::Future\\"\>Future\</a\> for \<a class=\\"struct\\" href=\\"https://docs.rs/tracing/0.1.41/x86\_64-unknown-linux-gnu/tracing/instrument/struct.Instrumented.html\\" title=\\"struct tracing::instrument::Instrumented\\"\>Instrumented\</a\>&lt;T&gt;\<div class=\\"where\\"\>where\\n T: \<a class=\\"trait\\" href=\\"https://doc.rust-lang.org/nightly/core/future/future/trait.Future.html\\" title=\\"trait core::future::future::Future\\"\>Future\</a\>,\</div\>\</div\>\<div class=\\"where\\"\> type \<a href=\\"https://doc.rust-lang.org/nightly/core/future/future/trait.Future.html#associatedtype.Output\\" class=\\"associatedtype\\"\>Output\</a\> = &lt;T as \<a class=\\"trait\\" href=\\"https://doc.rust-lang.org/nightly/core/future/future/trait.Future.html\\" title=\\"trait core::future::future::Future\\"\>Future\</a\>&gt;::\<a class=\\"associatedtype\\" href=\\"https://doc.rust-lang.org/nightly/core/future/future/trait.Future.html#associatedtype.Output\\" title=\\"type core::future::future::Future::Output\\"\>Output\</a\>;\</div\>","WithDispatch\<Self\>":"\<h3\>Notable traits for \<code\>\<a class=\\"struct\\" href=\\"https://docs.rs/tracing/0.1.41/x86\_64-unknown-linux-gnu/tracing/instrument/struct.WithDispatch.html\\" title=\\"struct tracing::instrument::WithDispatch\\"\>WithDispatch\</a\>&lt;T&gt;\</code\>\</h3\>\<pre\>\<code\>\<div class=\\"where\\"\>impl&lt;T&gt; \<a class=\\"trait\\" href=\\"https://doc.rust-lang.org/nightly/core/future/future/trait.Future.html\\" title=\\"trait core::future::future::Future\\"\>Future\</a\> for \<a class=\\"struct\\" href=\\"https://docs.rs/tracing/0.1.41/x86\_64-unknown-linux-gnu/tracing/instrument/struct.WithDispatch.html\\" title=\\"struct tracing::instrument::WithDispatch\\"\>WithDispatch\</a\>&lt;T&gt;\<div class=\\"where\\"\>where\\n T: \<a class=\\"trait\\" href=\\"https://doc.rust-lang.org/nightly/core/future/future/trait.Future.html\\" title=\\"trait core::future::future::Future\\"\>Future\</a\>,\</div\>\</div\>\<div class=\\"where\\"\> type \<a href=\\"https://doc.rust-lang.org/nightly/core/future/future/trait.Future.html#associatedtype.Output\\" class=\\"associatedtype\\"\>Output\</a\> = &lt;T as \<a class=\\"trait\\" href=\\"https://doc.rust-lang.org/nightly/core/future/future/trait.Future.html\\" title=\\"trait core::future::future::Future\\"\>Future\</a\>&gt;::\<a class=\\"associatedtype\\" href=\\"https://doc.rust-lang.org/nightly/core/future/future/trait.Future.html#associatedtype.Output\\" title=\\"type core::future::future::Future::Output\\"\>Output\</a\>;\</div\>"}

--- END RESULT ---

Any leftover html not caught by the htmltomd, maybe the links?

Even then, I doubt that would be enough for reasonably sized responses. If you were to parse out an actual hierarchy before the conversions, I imagine you could pick and choose what you respond with.

For example, if we parsed structs / methods / examples, etc, you could pass arguments for a "compact" mode that only does top level docs maybe. Thoughts?

[d6e]

Yeah i've been meaning to do that. I think actually just translating from html to markdown is maybe not a good idea. Since this is scraping a very specific site, I think maybe doing some custom parsing would be better. Ideally there shouldn't be any tokens that aren't necessary for the request