Add a maximum search results option

[michenriksen]

Hi there!

I work at GitLab, and we are using PageFind on our advisory database site. The search works great for terms like rails or flask, but we've noticed the page hangs when searching for something like cve-. This is likely because the index contains many results matching this query, as most advisories include a Common Vulnerability Enumeration ID (CVE-YYYY-DDDDD).

We believe a potential solution would be to implement a max_search_results option, which would stop the search after, say, 500 results.

Is it possible to add such an option?

Thank you for all the work on PageFind. It's an excellent search tool for large static sites! 🙇

[bglw]

👋 Hey @michenriksen

Thanks for reaching out — love the use case!

I can confirm I see the issue where the page hangs. I'll have a wee think about the best way to improve this, there's likely a few things that can be done. I'll train-of-thought some of it here:

---

First, ideally this search would be performant enough to run in its entirety — I'm a little surprised to see the hangup happening at this scale. A good reference would be the MDN Demo, where I don't get any noticeable hang when searching for the and getting 12,000 results. Similarly, this Twitter archive has no particular issue searching for t and getting all t* words.

My first suspicion for the hangup might be when calculating the filters, as my examples lack the amount of filtering being done in your case. Especially that Package filter, which looks particularly large. If that were the issue, improving the performance of filters would be my main priority, so that the search could retain its size.

My second suspicion is the amount of lookahead that might be happening. By default, a search for cve encompasses cve*, and here we see an outsized amount of unique words with that pattern (with some fun word-splitting ramifications too). If this was an issue, Pagefind could definitely add some sane limits here. (For example, if it finds >500 pages with cve, it could avoid looking at cve*)

Other performance things that have been on my list:

• (optionally) moving the search itself to a worker, to unblock the main thread. This wouldn't help with the delay, but would help with the jarring effect.
• some performance quirks I know could be improved for all cases, e.g., the WASM<>JS communication

Finally, we get to looking at something like max_search_results. My main reason for listing this last is that I'm unsure how best to go about it. I would be keen to implement it if I could offer some guarantee that it would still give you the 500 best results. Which depends where the big time sink currently is. If it's post-ranking, then that would be trivial, but if ranking is the issue then it might be harder to avoid it being 500 arbitrary results. I agree it's worth exploring though!

---

I'll build that site locally and try a few things out, first to see where the time is being lost and then what can be done to improve it 🙂

[michenriksen]

Wow, this is a remarkable level of support, thank you very much!

Please let me know if you need any help with building the site locally.

Just a heads-up: I added this kinda hacky measure in main.js yesterday which you would need to disable in your local build:

new PagefindUI({
  . . . 
    /**
     * HACK: in order to prevent the browser from freezing when terms have too
     * many results, we enforce that terms must have at least 4 characters, and
     * if the term looks like a search for a specific CVE, we require a year and
     * at least one digit of the sequence number.
     *
     * By returning an empty string, PageFind will not invoke the search backend.
     */
    processTerm: function (term) {
      term = String(term).trim();

      if (term.length < 4) {
        return "";
      }

      if (term.toLowerCase().startsWith("cve-")) {
        if (!term.match(/^cve-\d{4}-\d/i)) {
          return "";
        }
      }

      return term;
    },
  });

Thanks again for your help and interest in fixing this!