Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update redisearch base client to include timeout. Minor improvements on general latency and exported metrics #1

Open
wants to merge 183 commits into
base: master
Choose a base branch
from
Open

Update redisearch base client to include timeout. Minor improvements on general latency and exported metrics #1

wants to merge 183 commits into from

Conversation

filipecosta90
Copy link
Collaborator

No description provided.

filipecosta90 and others added 30 commits July 4, 2023 17:56
@filipecosta90
Update redisearch base client to include timeout. Extended latency me…
17ef49e 
…trics to include p50
@filipecosta90
Included batch_size and parallel info in upload results
1e8e719
@filipecosta90
Enable passing env variables for url and api_key qdrant parameters
084a68a
@filipecosta90
Print ef info if available in config
d890fa1
@filipecosta90
waiting on full index before return qdrant
dcdfe21
@filipecosta90
Revert "waiting on full index before return qdrant"
a1ba1b1 
This reverts commit dcdfe21.
@filipecosta90
Added REDIS_QUERY_TIMEOUT config parameter
87b0330
@filipecosta90
Introduced MILVUS_USER and MILVUS_PASS env vars
435038a
@filipecosta90
Introduce MILVUS_PORT env variable
f4e49be
@filipecosta90
Simplify milvus port settings
42133d0
@filipecosta90
Added missing milvus env variables on configure file
ee7cb9b
@filipecosta90
specify milvus uri when http is detected
dfefb49
@filipecosta90
wrap milvus upload_batch with backoff
9766207
@filipecosta90
Enabled using the API_KEY of weaviate and detecting secure connection…
9ad6022 
… setups
@filipecosta90
Only printing ef in search parameters if it's available
4e1408a
@filipecosta90
Simplified weviate client construct
824f17a
@filipecosta90
fix NameError on weviate search.py
31b2a18
@filipecosta90
Update weviate client to include multi-tenant schema check fix
909594d
@filipecosta90
Specify batch_size (and reduce from 64 to 32) on milvus configs.
42fc051
@filipecosta90
Updating milvus upload params from 16 to 4 concurrent clients doing b…
228b10d 
…atches of 32 vectors
@filipecosta90
Cleanup on milvus client
55f91c2
@alonre24
Change recall computation to be as ann-benchmarks'
9641080
@alonre24
Merge branch 'update.redisearch' of https://github.com/filipecosta90/…
666de49 
…vector-db-benchmark into update.redisearch
@filipecosta90
Reducing weaviate batch size due to recurrent ingest errors. ensuring…
e108874 
… cluster is ready/alive before benchmark
@filipecosta90
Reducing milvus batch size to ensure a steady ingestion
60360ad
@filipecosta90
Increasing timeout config for weaviate. Reduce parallel count on inge…
832fc86 
…stion to avoid bad state
@filipecosta90
Added single client weviate config
f8b4b7a
@filipecosta90
Added single client weviate config
c10df6d
@filipecosta90
Revert "Change recall computation to be as ann-benchmarks'"
32b4c6e 
This reverts commit 9641080.
@filipecosta90
Included single segment config for qdrant
0559dd4
filipecosta90 and others added 30 commits July 18, 2024 19:03
@filipecosta90
m-16-ef-64 for milvus
f5efe5b
@filipecosta90
Varying M/EF for intel/laion setup
a0c6ec8
@mpozniak95
Add REDIS_KEEP_DOCUMENTS and --only-configure flag
47c08c9
@filipecosta90
Merge pull request #6 from mpozniak95/add_1B_dataset
4a566f1 
Add laion-img-emb-768-1G-cosine to datasets
@filipecosta90
Added h5-multi file reader
4d1b39c
@filipecosta90
Fixed positional arg issue
7e140c5
@filipecosta90
Fixed some typos on loader script
2d8e250
@filipecosta90
using pipeline 1 and batch size 1 on CE upload
6f50710
@filipecosta90
Fixed laion-1b queries and dimension
f9e4e2d
@filipecosta90
control the number of concurrent screen sessions on the 1b loader
c0a5ad3
@filipecosta90
split hdf5 file into chunks for reads with less overhead
1448b72
@filipecosta90
5 clients per loader on CE loader
0b01eb4
@filipecosta90
Using 50 concurrent loaders on laion-1b
5e3b711
@filipecosta90
saving outputs of each loader
88552eb
@filipecosta90
Merge branch 'update.redisearch' into keep_documents
74d0cbb
@filipecosta90
Merge pull request #8 from mpozniak95/keep_documents
aa701aa 
Add REDIS_KEEP_DOCUMENTS and --only-configure flag
@filipecosta90
Adjust code for re-indexing
dca16d5
@filipecosta90
increase client count per runner
7d53dab
@filipecosta90
updated upload runner
23e2301
@filipecosta90
updated upload runner
38fb0c6
@filipecosta90
using hsetnx to speedup load in case of vector present
6ba28d4
@filipecosta90
logging missing keys
e3386be
@filipecosta90
Setting REDIS_KEEP_DOCUMENTS=1 as default
946ff60
@filipecosta90
Setting REDIS_KEEP_DOCUMENTS=1 as default
32b3843
@filipecosta90
Added MAX_QUERIES feature
2617cb4
@filipecosta90
fixed MAX_QUERIES
03b742b
@filipecosta90
Added REDIS_JUST_INDEX config to avoid resending duplicate data on th…
353e125 
…e wire
@mpozniak95
Add *args, **kwargs to read data method in reader interface (#10)
db1896d
@mpozniak95
If boto3 raises no credentials exception use urlib for downloading th…
2f9753a 
…e dataset (#11)
@mpozniak95
Update dockerfile and add single case scenario test (#12)
a47d6d3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@filipecosta90 @meiravgri @alonre24 @mpozniak95