Skip to content
This repository has been archived by the owner on Sep 3, 2024. It is now read-only.

Indexing logic improvements #6

Merged
merged 38 commits into from
Jun 14, 2024
Merged

Conversation

karol-t-wilk
Copy link
Collaborator

Sorry for the late and big PR, I will make sure the next ones are more manageable 😅

There are several improvements to the architecture of the app:

  • Multiple embedding models can be used at the same time
  • The app is now aware of packages and their releases
  • The app has support for future fragmentation of documentation
  • Adding a package to the app and embedding its docs are now separate actions, implemented as mix tasks
  • The demo page now lets you choose the embedding model used to perform the search
  • The Hex client uses only repo.hex.pm and does not touch the REST API

lib/search/packages.ex Outdated Show resolved Hide resolved
lib/mix/tasks/search/add.ex Outdated Show resolved Hide resolved
lib/mix/tasks/search/embed.ex Outdated Show resolved Hide resolved
lib/mix/tasks/search/embed.ex Outdated Show resolved Hide resolved
lib/search/embeddings/embedding.ex Outdated Show resolved Hide resolved
lib/search/packages.ex Outdated Show resolved Hide resolved
lib/search/hex_client.ex Outdated Show resolved Hide resolved
lib/search/hex_client.ex Outdated Show resolved Hide resolved
lib/search/hex_client.ex Outdated Show resolved Hide resolved
lib/search/packages.ex Outdated Show resolved Hide resolved
lib/search/packages.ex Outdated Show resolved Hide resolved
lib/search/packages.ex Outdated Show resolved Hide resolved
@karol-t-wilk karol-t-wilk marked this pull request as draft June 5, 2024 13:29
config/config.exs Outdated Show resolved Hide resolved
lib/search/packages.ex Outdated Show resolved Hide resolved
lib/search/packages.ex Outdated Show resolved Hide resolved
@karol-t-wilk karol-t-wilk marked this pull request as ready for review June 14, 2024 07:14
Copy link
Member

@wojtekmach wojtekmach left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me!

Comment on lines 48 to 50
else
{:error, _} = err ->
err
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this adds some strictness but I don't think it's worth it and so I'd just remove it.

Suggested change
else
{:error, _} = err ->
err

Comment on lines 56 to 57
else
err -> err
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this clause doesn't do anything, why do we keep it?

Comment on lines 70 to 72
else
{:error, _} = err ->
err
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto:

Suggested change
else
{:error, _} = err ->
err

Comment on lines 8 to 15
releases = [
%Release{package_name: "lesser", version: Version.parse!("1.2.3")},
%Release{package_name: "greater", version: Version.parse!("2.1.3")},
%Release{package_name: "middle", version: Version.parse!("1.4.3")}
]

assert %Release{package_name: "greater"} =
Release.latest(releases)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nitpick: this doesn't matter so I'm ok with keeping this as is but in practice we'd have the same package name but different version so I'd consider changing this and similar tests to be slightly more realistic:

Suggested change
releases = [
%Release{package_name: "lesser", version: Version.parse!("1.2.3")},
%Release{package_name: "greater", version: Version.parse!("2.1.3")},
%Release{package_name: "middle", version: Version.parse!("1.4.3")}
]
assert %Release{package_name: "greater"} =
Release.latest(releases)
releases = [
%Release{version: Version.parse!("1.2.3")},
%Release{version: Version.parse!("2.1.3")},
%Release{version: Version.parse!("1.4.3")}
]
assert Release.latest(releases).version == Version.parse!("2.1.3")

@karol-t-wilk karol-t-wilk merged commit 6350441 into main Jun 14, 2024
1 check passed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants