Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add basic query benchmark #46

Merged
merged 2 commits into from
Mar 12, 2024
Merged

Add basic query benchmark #46

merged 2 commits into from
Mar 12, 2024

Conversation

philippgille
Copy link
Owner

@philippgille philippgille commented Mar 12, 2024

  • Normalized vectors
  • Only query similarity
  • No metadata or content filter

With the current state of the project, not performance optimized except some parallelization with goroutines, on my first generation Framework Laptop 13, on performance mode "Balanced":

$ go test -benchmem -run=^$ -bench .
goos: linux
goarch: amd64
pkg: github.com/philippgille/chromem-go
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
BenchmarkCollection_Query_NoContent_100-8      	    3082	    401136 ns/op	 1240100 B/op	     238 allocs/op
BenchmarkCollection_Query_NoContent_1000-8     	     435	   2735553 ns/op	12372113 B/op	    2038 allocs/op
BenchmarkCollection_Query_NoContent_5000-8     	      79	  12975637 ns/op	61843650 B/op	   10039 allocs/op
BenchmarkCollection_Query_NoContent_25000-8    	      18	  65804944 ns/op	309209240 B/op	   50038 allocs/op
BenchmarkCollection_Query_NoContent_100000-8   	       4	 279518934 ns/op	1236805992 B/op	  200041 allocs/op
BenchmarkCollection_Query_100-8                	    2860	    414340 ns/op	 1240069 B/op	     238 allocs/op
BenchmarkCollection_Query_1000-8               	     444	   2770227 ns/op	12372085 B/op	    2038 allocs/op
BenchmarkCollection_Query_5000-8               	      79	  12908316 ns/op	61843567 B/op	   10038 allocs/op
BenchmarkCollection_Query_25000-8              	      15	  67794886 ns/op	309209185 B/op	   50038 allocs/op
BenchmarkCollection_Query_100000-8             	       4	 287507129 ns/op	1236805700 B/op	  200038 allocs/op
PASS
ok  	github.com/philippgille/chromem-go	31.496s

This is to establish a baseline, from which we can start optimizing while making sure nothing that one might assume is an optimization is actually making things worse. For example passing thousands of values through a channel can lead to a lot of data copying, and locking with a mutex can actually be faster. (Example from a recent One Billion Row Challenge (in Go) article)

- Normalized vectors
- Only query similarity
- No metadata or content filter
- More relevant for basic chromem-go usage
- String copying during querying might be very relevant for performance
@philippgille philippgille merged commit f59e9dc into main Mar 12, 2024
2 checks passed
@philippgille philippgille deleted the add-query-benchmark branch March 12, 2024 21:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant