Search through OCaml Documentation

[cuihtlauac]

@patricoferris has implemented an in-memory search engine

See:
https://discuss.ocaml.org/t/ann-ocaml-search-0-1-1-in-memory-search-index-for-ocaml-values/11179

https://github.com/patricoferris/ocaml-search

Seems a perfect fit for ocaml.org

[tmattio]

Maybe rename "Search through OCaml Documentation" or something? ocaml-search looks like a fantastic library (thanks @patricoferris!) but the end goal is to search through the doc, not integrate the library per say.

[sabine]

We have these places where we need search:

Implemented:

• packages (server-side using string-contains)
• papers (server-side using string-contains)
• releases (server-side using string-contains)
• academic users (server-side using string-contains)
• symbols in stdlib API (frontend-only JS search)

Unimplemented:

• symbols in single package documentation
• full-text in single package documentation
• symbols declared across all opam packages
• full-text in learn area
• full-text in ocaml manual

Did I forget any places/purposes we need search?

There's two potential ways we could use https://github.com/patricoferris/ocaml-search:

• via js_of_ocaml to provide frontend-only search (though, we could also use the JS-library he took as inspiration directly)
• server-side, as long as we can fit the data in memory

[yawaramin]

Are we talking about the existing search box on https://ocaml.org/, which currently says 'Search OCaml Packages'? In other words, are we planning to make that a universal search box for all of the above-mentioned resources?

Does this ticket relate to #11 and #665 and if so, should they all be merged into a single ticket?

When you say 'Implemented' for some of the above resources, for packages I understand how to use the search box, but how can I search for papers, releases, etc.? [EDIT: found the search boxes on the individual pages, please ignore this question]

ocaml-search looks cool but the readme says: 'It is not particularly optimised, memory efficient or tested. Use with care!' Are we sure we want to use this instead of, say, SQLite's fulltext search which looks like it can cover many of the above fulltext search needs? EDIT: SQLite's FTS5 would typically be used in the backend, and with an SQLite database stored in a file.

[cuihtlauac]

We've not yet reached a consensus on search for ocaml.org. We want ocaml.org to be the best source of information on OCaml. In this perspective, there's a high-level agreement that more search capability is a requirement. However, neither the scope nor the implementation options are decided. On the constraint side, we'd like to build on the existing code as much as possible.

Update:
Shercode and Sherlodoc are super useful (and cool). We did figure out how ocaml.org could benefit from them.

[patricoferris]

Agree with pretty much everything that has been said here. ocaml-search is probably not what you want in order to search through OCaml documentation, it can't persist indices yet and can only work with things in-memory at the moment. If for example you want to autocomplete dune package names directly in the search box, then it could be used for that kind of thing.

@art-w's excellent https://github.com/art-w/sherlocode is the way to go I think for this sort of thing :))

[yawaramin]

OK, if anyone is interested in a demo of SQLite's FTS capabilities, let me know. I built a Dream app powered by it for myself and a few friends and it's working out really nicely. Search is fast and results are extremely relevant.