Add Typesense #44

ruslandoga · 2024-09-27T13:44:14Z

This PR integrates Typesense into Hexdocs.

TODOs:

should Broadway task fail on failed indexing, or should it just log an error?
Logging excessively for now
should indexing happen in the same step as upload to object storage or can it be parallel?
Indexing in the same step for now
index proglang
index Erlang, what's its search_data equivalent?
The format is the same: Add Typesense #44 (comment)
index Gleam: ce7af3b
Commit reverted, approach needs discussion with Gleam team
more tests

CI results: ruslandoga#1

lib/hexdocs/search.ex

test/hexdocs/queue_test.exs

lib/hexdocs/queue.ex

lib/hexdocs/search.ex

josevalim · 2024-09-29T21:10:19Z

Thank you @ruslandoga! I am currently on holidays but I will try to carve some time sooner than later to give you feedback. /cc @wojtekmach

josevalim · 2024-09-29T21:13:46Z

index erlang, what's its search_data equivalent?

Erlang uses ExDoc now, so they have the exact same structure. However, we will need to either poll them or ask them to ping us once they publish a new version or ask them to push their docs to Hexdocs! I'd say we can postpone this to a follow up pull request.

On the other hand, I believe Gleam does not have the data in the format we need (we created our search_data.js somewhat recently thinking exactly about this). They would need to update their tools first, so we can postpone this conversation too.

wojtekmach · 2024-10-01T08:49:15Z

@ruslandoga I'll try to review this sooner than later but I have a lot on my plate until October 15th, after that it's gonna be my number one priority to see this through. Sorry about that.

lib/hexdocs/queue.ex

lib/hexdocs/search.ex

wojtekmach

This is very exiting! I left some comments below.

test/test_helper.exs

lib/hexdocs/search/typesense.ex

lib/hexdocs/queue.ex

ruslandoga · 2024-11-10T13:57:46Z

👋

I think it's ready for review now :)

Updates since the last review:

removed Gleam indexing (for now)
added proglang field to the schema
moved some code around in attempt to make it cleaner
added more tests

ericmj · 2024-11-12T17:37:40Z

lib/hexdocs/search/search.ex

+    search_data_js =
+      Enum.find_value(files, fn {path, content} ->
+        case Path.basename(path) do
+          "search_data-" <> _digest -> content


We cannot trust this data since it's user provided, can they do anything dangerous by providing something we don't expect? Maybe we should do some rudimentary validation?

They can provide long strings like https://github.com/cloudpods-dev/docker-engine-api-elixir/blob/813cc557da483f623a8f484db04efc7e58db0376/lib/docker_engine_api/api/container.ex#L67, but Typesense seems to handle it fine. We can check for content size, maybe. I think if Typesense doesn't like the payload, it would simply reject it.

I added a test that checks that invalid fields in search items (like type being a map instead of a string, or doc being a list) are rejected: 8d58e4f

Co-authored-by: Eric Meadows-Jönsson <[email protected]>

ruslandoga · 2024-11-13T05:49:47Z

config/runtime.exs

@@ -5,6 +5,9 @@ if config_env() == :prod do
    port: System.fetch_env!("HEXDOCS_PORT"),
    hexpm_url: System.fetch_env!("HEXDOCS_HEXPM_URL"),
    hexpm_secret: System.fetch_env!("HEXDOCS_HEXPM_SECRET"),
+    typesense_url: System.fetch_env!("TYPESENSE_URL"),
+    typesense_api_key: System.fetch_env!("TYPESENSE_API_KEY"),
+    typesense_collection: System.fetch_env!("TYPESENSE_COLLECTION"),


I created hexdocs-test collection on Typesense Cloud, https://cloud.typesense.org/clusters/ent97o5sv4dzx2f0p/collections

I think we can use it during alpha/beta testing.

ruslandoga · 2024-11-13T05:51:54Z

config/runtime.exs

@@ -5,6 +5,9 @@ if config_env() == :prod do
    port: System.fetch_env!("HEXDOCS_PORT"),
    hexpm_url: System.fetch_env!("HEXDOCS_HEXPM_URL"),
    hexpm_secret: System.fetch_env!("HEXDOCS_HEXPM_SECRET"),
+    typesense_url: System.fetch_env!("TYPESENSE_URL"),
+    typesense_api_key: System.fetch_env!("TYPESENSE_API_KEY"),


This would be the "Admin API Key" for https://cloud.typesense.org/clusters/ent97o5sv4dzx2f0p cluster, it can be downloaded from the dashboard.

ruslandoga · 2024-11-13T05:52:30Z

config/runtime.exs

@@ -5,6 +5,9 @@ if config_env() == :prod do
    port: System.fetch_env!("HEXDOCS_PORT"),
    hexpm_url: System.fetch_env!("HEXDOCS_HEXPM_URL"),
    hexpm_secret: System.fetch_env!("HEXDOCS_HEXPM_SECRET"),
+    typesense_url: System.fetch_env!("TYPESENSE_URL"),


And this would be https://ent97o5sv4dzx2f0p.a1.typesense.net

ruslandoga · 2024-11-18T06:06:28Z

👋

Just wanted to check if there’s anything else we'd need to address before it's merged?

wojtekmach

Looks good to me. We plan to test this out in our staging server before merging. Thank you for all the work!

This reverts commit 26169bb.

wojtekmach · 2024-12-12T13:45:56Z

@ruslandoga this is now deployed to https://staging.hexdocs.pm and I have published a test package and it was correctly indexed into hexdocs-test collection.

Is this ready to publish to prod? What are the next steps?

Thank you for all of your work on this and apologies for delays.

ruslandoga · 2024-12-12T13:51:19Z

👋 @wojtekmach

Yes, I think it's ready for prod! I think the next step would be integrating global search into ex_doc. I can open a PR!

ruslandoga · 2024-12-12T13:53:45Z

We can create a new collection if needed or we can continue using hexdocs-test

I think it's possible to clone / fork collections in Typesense cloud, and if not, it's pretty easy to move the data around, since I guess by the nature of Hexdocs, the indexed data is immutable. So it should be OK either way.

wojtekmach · 2024-12-12T13:55:19Z

Got it, excellent!

Could you create hexdocs-staging and hexdocs-prod collections for us? We tend to follow that particular convention for external services.

Should we backfill some data? Latest versions of all packages? All versions of all packages? We can also do nothing for now and make a decision when all pieces are ready. cc @josevalim

ruslandoga · 2024-12-12T14:05:51Z

I've created hexdocs-staging and hexdocs-prod collections with the same schema.

Regarding backfilling, it definitely helps during development but I tend to use a local Typesense instance. It's a bit outdated by now (e.g. missing proglang), but here're some docs I used before: https://hexdocs-artifacts.s3.eu-central-003.backblazeb2.com/docs_from_tarballs_all_versions.jsonl.zst (it's 190MB compressed, 4.4G uncompressed)

$ curl https://hexdocs-artifacts.s3.eu-central-003.backblazeb2.com/docs_from_tarballs_all_versions.jsonl.zst -O
$ zstd docs_from_tarballs_all_versions.jsonl.zst -d
# add proglang=elixir to all entries
$ jq '. + {proglang: "elixir"}' docs_from_tarballs_all_versions.jsonl > docs.jsonl
$ docker compose up typesense -d

# https://typesense.org/docs/27.1/api/collections.html#with-pre-defined-schema
$ curl "http://localhost:8108/collections" \
       -X POST \
       -H "Content-Type: application/json" \
       -H "X-TYPESENSE-API-KEY: hexdocs" \
       -d '{"fields": [
    {"facet": true, "name": "proglang", "type": "string"},
    {"facet": true, "name": "type", "type": "string"},
    {"name": "title", "type": "string"},
    {"name": "doc", "type": "string"},
    {"facet": true, "name": "package", "type": "string"}
  ],
  "name": "hexdocs-local",
  "token_separators": [".", "_", "-", " ", ":", "@", "/"]
}'

# https://typesense.org/docs/27.1/api/documents.html#import-a-jsonl-file
$ curl "http://localhost:8108/collections/hexdocs-local/documents/import?action=create" \
       -X POST \
       -T docs.jsonl \
       -H "X-TYPESENSE-API-KEY: hexdocs"

# sanity check
$ curl -H "X-TYPESENSE-API-KEY: hexdocs" "http://localhost:8108/collections/hexdocs-local/documents/777777"

josevalim · 2024-12-12T20:40:30Z

@ericmj @wojtekmach @ruslandoga let's definitely backfill. We only support recent ExDoc versions anyway, which will act as a filter.

A couple things to figure out:

Will Elixir releases be indexed automatically since we push to hexdocs.pm or does it require special treatment?
We should do something for Erlang in particular
Are we going to make typesense address public or are we going to expose it through hex.pm and use fast.ly to rewrite to typesense?

Overall, my next step suggestion is to build a home page for searching within a given set of packages. This thread on ElixirForum has a good example: https://elixirforum.com/t/hexdocs-search-engine-for-us-devs/46814/1

We could use it at least in two places:

A new hexdocs.pm home? We can store the selected packages in your cookies
mix hex.search which opens up a page with all of the hex packages of your lock file filled in (similar to mix hex.outdated). We can rename the existing mix hex.search to mix hex.find

For ExDoc, we would need to work on the related packages feature: elixir-lang/ex_doc#1811

My idea is that we would be able to store this information as a .json file as well. So phoenix_html can say related_deps: :phoenix and we just query https://hexdocs.pm/phoenix/related_deps.json

josevalim · 2024-12-13T11:08:13Z

I opened new issues: #46 #47 #48 #49.

josevalim · 2024-12-13T13:39:56Z

This one looks good to merge to me. We can continue in the other issues.

wojtekmach · 2024-12-13T13:53:05Z

Sounds good, I'll finish infrastructure setup and deploy this to prod soon.

wojtekmach · 2024-12-13T15:19:11Z

This is now running on prod, so far so good!

josevalim · 2024-12-13T16:07:55Z

Amazing work @ruslandoga !

ruslandoga force-pushed the add-typesense branch from ca70475 to 859dd13 Compare September 27, 2024 13:52

ruslandoga mentioned this pull request Sep 27, 2024

add typesense ruslandoga/hexdocs#1

Draft

ruslandoga commented Sep 27, 2024

View reviewed changes

lib/hexdocs/search.ex Outdated Show resolved Hide resolved

ruslandoga commented Sep 27, 2024

View reviewed changes

test/hexdocs/queue_test.exs Outdated Show resolved Hide resolved

ruslandoga commented Sep 27, 2024

View reviewed changes

lib/hexdocs/queue.ex Outdated Show resolved Hide resolved

ruslandoga commented Sep 27, 2024

View reviewed changes

lib/hexdocs/search.ex Outdated Show resolved Hide resolved

ericmj reviewed Oct 2, 2024

View reviewed changes

lib/hexdocs/queue.ex Outdated Show resolved Hide resolved

lib/hexdocs/queue.ex Outdated Show resolved Hide resolved

lib/hexdocs/search.ex Outdated Show resolved Hide resolved

lib/hexdocs/search.ex Outdated Show resolved Hide resolved

wojtekmach reviewed Oct 18, 2024

View reviewed changes

josevalim reviewed Oct 21, 2024

View reviewed changes

lib/hexdocs/queue.ex Outdated Show resolved Hide resolved

josevalim reviewed Oct 21, 2024

View reviewed changes

lib/hexdocs/queue.ex Outdated Show resolved Hide resolved

josevalim reviewed Oct 21, 2024

View reviewed changes

lib/hexdocs/queue.ex Outdated Show resolved Hide resolved

josevalim reviewed Oct 21, 2024

View reviewed changes

lib/hexdocs/queue.ex Outdated Show resolved Hide resolved

ruslandoga mentioned this pull request Oct 21, 2024

Update runtime #45

Merged

ruslandoga force-pushed the add-typesense branch from ce7af3b to e18e317 Compare November 10, 2024 06:42

ruslandoga marked this pull request as ready for review November 10, 2024 13:57

ruslandoga mentioned this pull request Nov 10, 2024

Hexdocs indexing Gleam packages gleam-lang/gleam#3811

Open

ericmj approved these changes Nov 12, 2024

View reviewed changes

ruslandoga and others added 9 commits November 13, 2024 12:32

add typesense

08e7b64

fix ci

b59abed

Update lib/hexdocs/queue.ex

79edf7d

Co-authored-by: Eric Meadows-Jönsson <[email protected]>

add compose

67ae461

hide typesense behind impl

4eecb02

simplify find_search_items/3

1c727da

cleanup test

2ed6007

include typesense in ci tests

f0a39a0

add post in hexdocs.http

649397a

ruslandoga added 3 commits November 13, 2024 12:32

remove search_data_js format error logs when there is no search_data_js

d7cb3b9

rm releases.exs

dd222ea

read Typesense collection from env

bfc0539

ruslandoga force-pushed the add-typesense branch from 5d9f67d to bfc0539 Compare November 13, 2024 05:34

ruslandoga added 3 commits November 13, 2024 12:36

switch to :json

1a97386

update tests

1087e6a

add tests for invalid search items

8d58e4f

ruslandoga commented Nov 13, 2024

View reviewed changes

josevalim approved these changes Nov 13, 2024

View reviewed changes

wojtekmach approved these changes Nov 18, 2024

View reviewed changes

wojtekmach added 2 commits December 11, 2024 12:31

Merge branch 'main' into add-typesense

9a839e4

Prepare hexdocs staging for typesense

a4ba5c3

wojtekmach force-pushed the add-typesense branch from 4c553de to a4ba5c3 Compare December 11, 2024 12:26

wojtekmach added 2 commits December 11, 2024 21:59

wip

26169bb

Revert "wip"

ff16251

This reverts commit 26169bb.

wojtekmach merged commit b1f48e9 into hexpm:main Dec 13, 2024
3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Typesense #44

Add Typesense #44

ruslandoga commented Sep 27, 2024 •

edited

Loading

josevalim commented Sep 29, 2024

josevalim commented Sep 29, 2024

wojtekmach commented Oct 1, 2024

wojtekmach left a comment

ruslandoga commented Nov 10, 2024 •

edited

Loading

ericmj Nov 12, 2024

ruslandoga Nov 12, 2024 •

edited

Loading

ruslandoga Nov 13, 2024 •

edited

Loading

ruslandoga Nov 13, 2024 •

edited

Loading

ruslandoga Nov 13, 2024 •

edited

Loading

ruslandoga Nov 13, 2024 •

edited

Loading

ruslandoga commented Nov 18, 2024 •

edited

Loading

wojtekmach left a comment

wojtekmach commented Dec 12, 2024

ruslandoga commented Dec 12, 2024 •

edited

Loading

ruslandoga commented Dec 12, 2024 •

edited

Loading

wojtekmach commented Dec 12, 2024

ruslandoga commented Dec 12, 2024 •

edited

Loading

josevalim commented Dec 12, 2024

josevalim commented Dec 13, 2024

josevalim commented Dec 13, 2024

wojtekmach commented Dec 13, 2024

wojtekmach commented Dec 13, 2024

josevalim commented Dec 13, 2024

Add Typesense #44

Add Typesense #44

Conversation

ruslandoga commented Sep 27, 2024 • edited Loading

josevalim commented Sep 29, 2024

josevalim commented Sep 29, 2024

wojtekmach commented Oct 1, 2024

wojtekmach left a comment

Choose a reason for hiding this comment

ruslandoga commented Nov 10, 2024 • edited Loading

ericmj Nov 12, 2024

Choose a reason for hiding this comment

ruslandoga Nov 12, 2024 • edited Loading

Choose a reason for hiding this comment

ruslandoga Nov 13, 2024 • edited Loading

Choose a reason for hiding this comment

ruslandoga Nov 13, 2024 • edited Loading

Choose a reason for hiding this comment

ruslandoga Nov 13, 2024 • edited Loading

Choose a reason for hiding this comment

ruslandoga Nov 13, 2024 • edited Loading

Choose a reason for hiding this comment

ruslandoga commented Nov 18, 2024 • edited Loading

wojtekmach left a comment

Choose a reason for hiding this comment

wojtekmach commented Dec 12, 2024

ruslandoga commented Dec 12, 2024 • edited Loading

ruslandoga commented Dec 12, 2024 • edited Loading

wojtekmach commented Dec 12, 2024

ruslandoga commented Dec 12, 2024 • edited Loading

josevalim commented Dec 12, 2024

josevalim commented Dec 13, 2024

josevalim commented Dec 13, 2024

wojtekmach commented Dec 13, 2024

wojtekmach commented Dec 13, 2024

josevalim commented Dec 13, 2024

ruslandoga commented Sep 27, 2024 •

edited

Loading

ruslandoga commented Nov 10, 2024 •

edited

Loading

ruslandoga Nov 12, 2024 •

edited

Loading

ruslandoga Nov 13, 2024 •

edited

Loading

ruslandoga Nov 13, 2024 •

edited

Loading

ruslandoga Nov 13, 2024 •

edited

Loading

ruslandoga Nov 13, 2024 •

edited

Loading

ruslandoga commented Nov 18, 2024 •

edited

Loading

ruslandoga commented Dec 12, 2024 •

edited

Loading

ruslandoga commented Dec 12, 2024 •

edited

Loading

ruslandoga commented Dec 12, 2024 •

edited

Loading