The search index is a bit large on the JAM stack SSG

[patrickdung]

Hello,

Thanks for the great project first.
I am using Pelican, so I use this plugin.
I found the search-index is a bit large. It's about 375KB.
I was using Tipue, the JSON index size is about half of the search index by Stork.

The index size is larger than the HTML size of my article (each article is about 40KB). So the user / client side has to download quite a large index. I am curious if the index could be further reduced. I am not sure if it's a Stork problem or the plugin problem.
If you want to look at the index, it's in here, thanks!

Edit: I am able to use server side Brotli compression to make the search index to transfer as small as about 96KB.

[jameslittle230]

Index size is definitely a priority for the project, and I agree that there's more that I could do to improve it! I think the reason it's noticably large today is because I've prioritized search bar usability (i.e. getting as many relevant results as possible and showing them with contextual snippets) over pure index size reductions.

Here are a few projects in this area that I want to tackle:

• Compression helps a lot, as you've found, signifying that there's a lot of repetition in the binary index format. Maybe I could build in some of that compression in the next index format.
• Could Stork only index the top n "unique" words in the corpus, so you still get meaningful results but without a lot of common-language boilerplate?
• Stork includes a full copy of the document text so it can create excerpts that show your search term in context. I'd like to add a config that removes the excerpt feature by omitting those full copies of the document.

I'd definitely appreciate any other suggestions or feedback you have.

Thanks for using Stork!
-James

[patrickdung]

Thanks for reply.

I create blog postings so my viewpoint is mostly from a user perspective.

• Could Stork only index the top n "unique" words in the corpus, so you still get meaningful results but without a lot of common-language boilerplate?

Sorry I can't get the meaning of this item, please give a simple example.

• Stork includes a full copy of the document text so it can create excerpts that show your search term in context. I'd like to add a config that removes the excerpt feature by omitting those full copies of the document.

So it needs to trim down what to index. I am using Pelican SSG and have an observation:

• There is a Pelican Plugin called Tipue. It indexes the website as a JSON file.
  This website is using it, he has more articles than my website.
  The index size is about 87KB gzipped, 255KB plaintext.

[justinmayer]

Hey folks. I am Pelican's dev team lead, and I wrote the Pelican Search Plugin. I think the first and easiest step would be to add documentation to Stork about configuring web servers so that search indexes can be served in compressed form (for example, via Brotli, as mentioned in this thread).

Comparing to Tipue is only partially useful. Aside from being a completely dead project, Tipue also has less functionality than Stork, as James points out in this thread. So to some degree, it's understandable that there would be an index size difference.

That said, James suggested some interesting ideas regarding how the index size might be reduced, and those ideas could indeed be useful as optional (not default) configuration settings. 👍

[patrickdung]

Here's the relevant config for server side Brotli.
Append/update netlify.toml:

[[headers]]
  for = "/search-index.st"
  [headers.values]
    Content-Type = "application/wasm"
    Cache-Control = "max-age=43200, must-revalidate"

I use this is for Pelican Search Plugin and Netlify.
I choose the content type as application/wasm and it seems to work.
Also the user should update the cache-control settings accordingly.

[ndeville]

Reviving this thread as my index file is now 13Mb (with 470 articles generated by Pelican) and impacts heavily usability - on mobile especially, the page loads only halfway and waits until the 13Mb file has been downloaded until the rest of the article is displayed.

My site is deployed on Github Pages and I couldn't find if/how to compress the file server-side.

Has there been any development re index compression, or any other workaround I should use?

[ndeville]

I managed to decrease the index file to 9Mb by modifying the Stork settings within the Pelican Search plugin as follows:

    search_settings = cleandoc(
        f"""
            [input]
            base_directory = "{base_dir}"
            html_selector = "{self.html_selector}"

            [output]
            excerpts_per_result = 0
            {input_files}
        """
    )

However... no results anymore in the search UI! ("No files found", no matter the input)

I reverted to the standard config and the 13Mb index file for now.

Site is at https://notes.nicolasdeville.com/ FYI

[Bobspadger]

Are there any more suggestions on this, I'd love to re-implement search on my site, but the current index is 45MB, which is a little bit too large to make it useful.

[hrw]

For my website main page has 258KB in total (uncompressed). Stork adds 420KB of js/wasm/css.

And 8.2MB of index file. Which even with gzip compression is 2-2.5MB.