Search functionality smarter than good

[marcusmueller]

Problem

You search for a term like boost.url.

You're presented with multiple pages of results containing substrings of the query.

You search for rapidcsv. Instead of a simple "nope, sorry", you get an assorted list things that match substrings of the query.

Proposed solution

Don't be smart. Search for exact whole-word matches only. If multiple words are entered in the search box, only return results that match all of the terms as exact whole-word match.

If current functionality is desired by users (I have my doubts), then clearly separate "exact matches" from "partial matches".

[hobofan]

Yes, search is notoriously hard, and not a lot of work has gone into optimizing the search on the site.

The examples you present in the "Problem" section do not look like problems from my point of view. The goal of search is not to provide "only correct results", but to "surface the most relevant results in the best priority". If that is done well, the inference of "nope, sorry" can trivially be made by the user if the top (or top few) results is not the one they were looking for.
I think realistically we have much more of a problem with not being able to surface relevant results rather than surfacing too many results (see #39). This is mostly due to the BCR registry data being incredibly sparse in searchable data (we only have the module name to go off) and data from e.g. Github being hard to accumulate and keeping it up-to-date.

In reality, users are likely to not know the exact search term, and are even more likely to make typos. E.g. they will search for boost_url instead of the correct boost.url, or typo it to boost.ulr. If we were to do exact matches, the result would be fully empty, leaving the user at a dead end.

What the BCR registry site uses is fuzzy search. This eliminates the whole class of typo errors without much implementation effort, and still prioritizes exact matches to the top (see this article for a bit more of an introduction). I would rather call this "using an industry proven solution" rather than "trying to be smart".

[marcusmueller]

Hm, in that case, I might have misunderstood who should be using that search – not me :)
I want to really see the things that match, not thrown together with "kind of also close" things that aren't at all close (rapidjson is not in any way related to rapidcsv, for example, and there's no way that someone searching for one would want the other ;) ).

But I think the statement here is "OK, you're too precise, go and check your own checkout of bazel-central-registry instead of asking a web frontend, if you want string-perfect matchs"; which is fine for, say, a sports good shop, but for "might have catastrophic security implication if using a wordtwist / typo-squatter replacement" thing like a software repo? I don't know… I might be a bit old-school in that, though.

[hobofan]

I don't think it's wrong to want to have a frontend that also supports exact string matches. I just wanted to elaborate why bcr-ui is doing what it's doing and why that may not match your expectations. Almost every other frontend for a package manager registry I am aware of implements more sophisticated search (usually with BM25 and an actual search backend server, which we don't have) that comes closer to fuzzy search than exact matches.

Two options that I could see working:

• Opt-in exact search by providing quotes around the whole query string (as Google supported once upon a time)
• Showing an indication that no exact match was found followed by a list of approximate matches, as e.g. lib.rs does here: https://lib.rs/search?q=syrde. I don't know the exact logic they use for showing/not showing that indicator though, as just cutting of the last character doesn't result in that indicator: https://lib.rs/search?q=serd

---

but for "might have catastrophic security implication if using a wordtwist / typo-squatter replacement" thing like a software repo? I don't know… I might be a bit old-school in that, though.

I very much get that concern. However realistically, the security model of the bcr-ui very much relies on the data that goes into the BCR. That may or may not appeal to your old-school sensibilities, as it involves manual vetting of submissions (where typo-squatting and maintainership changes are hard to sneak by) ;). While of course still susceptible to human error, that also eliminates most package manager attack vectors that rely on automation.

[marcusmueller]

:) Thanks for the kind words!

Opt-in exact search by providing quotes around the whole query string (as Google supported once upon a time)

sounds like an increase of search implementation complexity, but generally like something that I'd want!

While of course still susceptible to human error, that also eliminates most package manager attack vectors that rely on automation.

You're right about that; hm. I'd then conclude that improving this is nice-to-have, but not high-prio

[alexeagle]

Sounds like this was resolved, in the sense that yes, we want a good industry-standard search solution. I'll take a look at adding Algolia Docsearch which is free for open-source projects.

[alexeagle]

@hobofan I hooked this up.

https://dashboard.algolia.com/apps/1OP3OUPBDN/crawler/crawler/3ce256d3-41d6-44a6-85dc-4fbbf2599fa6/overview is the config for the crawler, which lets us rank pages (for example, higher ranking for literal ruleset name)
https://legendary-waffle-xg7qpj49cvx5v-3000.app.github.dev/ is serving the app with the new search box (at least as long as GitHub leaves it running... we ought to serve previews of PRs)

[alexeagle]

@hobofan @adincebic I think we need to generate a sitemap.xml file that has only the top-level module pages, so the crawler can index latest-version-only. Have time for that?

[hobofan]

@alexeagle I added the pattern negative pattern !https://registry.bazel.build/modules/*/* to the crawler config (and tested with the testing tool), which should have the same effect as creating a sitemap specifically for that.

[alexeagle]

Yup, that works - next issue with indexing is that the Description for each module is rendered client-side. @adincebic is working on that.

WDYT about merging #176 to switch over to Docsearch now? Or wait until it has more benefit?

[alexeagle]

@hobofan I'm also looking into extending/improving the existing search indexing/query rather than adding a third-party service.

I see https://www.fusejs.io/api/options.html is a way we can tune the results. Have you already spent some time investigating what's possible here?

[hobofan]

I see fusejs.io/api/options.html is a way we can tune the results. Have you already spent some time investigating what's possible here?

No, I haven't deeply looked into it.

Another solution that I've encountered more and more recently is https://pagefind.app/ , which now with the full static rendering is more of an option. The configuration there would be more similar to how we would configure Algolia, but also with options to e.g. weight individual sections (https://pagefind.app/docs/weighting/#ranking-content-higher-or-lower) of the page differently.

[alexeagle]

Yes I can see how that one is appealing here. Simple, no external site to hold credentials and config, probably not hard to integrate in our build pipeline.

I'm going to work on getting starlark API docs rendered, so I'm not taking action on this anytime soon. @hobofan are you interested to make the changes?