Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

support basic searcher #351

Merged
merged 1 commit into from
Jan 24, 2025
Merged

support basic searcher #351

merged 1 commit into from
Jan 24, 2025

Conversation

ShawnShawnYou
Copy link
Collaborator

LHT129
LHT129 reviewed Jan 20, 2025
View reviewed changes
src/allocator_wrapper.h Outdated
template <typename T, typename U>
bool
operator==(const AllocatorWrapper<T>&, const AllocatorWrapper<U>&) noexcept {
return true;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

always true？

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed

LHT129
LHT129 reviewed Jan 20, 2025
View reviewed changes
src/data_cell/flatten_datacell.h Outdated
@@ -70,6 +70,11 @@ class FlattenDataCell : public FlattenInterface {
io_->Prefetch(id * code_size_);
};

bool
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not used here

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

LHT129
LHT129 reviewed Jan 20, 2025
View reviewed changes
src/impl/basic_searcher.cpp Outdated

template <typename GraphTmpl, typename VectorDataTmpl>
BasicSearcher<GraphTmpl, VectorDataTmpl>::BasicSearcher(std::shared_ptr<GraphTmpl> graph,
std::shared_ptr<VectorDataTmpl> vector,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use interface is better

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

LHT129
LHT129 reviewed Jan 20, 2025
View reviewed changes
src/impl/basic_searcher.cpp Outdated Show resolved Hide resolved
LHT129
LHT129 reviewed Jan 20, 2025
View reviewed changes
src/impl/basic_searcher.cpp Outdated
for (uint32_t i = 0; i < neighbors.size(); i++) {
#ifdef USE_SSE
if (i + prefetch_neighbor_visit_num_ < neighbors.size()) {
_mm_prefetch(vl->mass + neighbors[i + prefetch_neighbor_visit_num_], _MM_HINT_T0);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

inabao
inabao reviewed Jan 20, 2025
View reviewed changes
src/impl/basic_searcher.cpp Outdated Show resolved Hide resolved
src/impl/basic_searcher.h Outdated
#include <shared_mutex>

#include "../utils.h"
#include "ThreadPool.h"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove unused header files

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

src/impl/basic_searcher.h Outdated Show resolved Hide resolved
src/impl/runtime_parameter.h Outdated
#include <string>
#include <thread>
#include <variant>
#include <vector>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

src/impl/basic_searcher_test.cpp
#include "fixtures.h"
#include "io/memory_io.h"
#include "quantization/fp32_quantizer.h"
#include "safe_allocator.h"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

src/impl/basic_searcher_test.cpp Outdated
REQUIRE(result.top().second == valid_result.top().second);
REQUIRE(result.top().second == ids[i]);
}
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add new line

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

src/impl/runtime_parameter.h Outdated
int cur_{0};
bool is_end_{false};
};
} // namespace vsag
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add new line

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

src/impl/basic_searcher.h Outdated
uint32_t prefetch_neighbor_visit_num_{1};
};

} // namespace vsag
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add new line

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

src/impl/basic_searcher.cpp Outdated
}

MaxHeap
BasicSearcher::Search(const float* query, InnerSearchParam& inner_search_param) const {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

const InnerSearchParam& inner_search_param

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

src/data_cell/flatten_datacell.h Outdated
@@ -226,7 +226,18 @@ FlattenDataCell<QuantTmpl, IOTmpl>::query(float* result_dists,
const std::shared_ptr<Computer<QuantTmpl>>& computer,
const InnerIdType* idx,
InnerIdType id_count) {
for (uint32_t i = 0; i < this->prefetch_neighbor_codes_num_ and i < id_count; i++) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

prefetch_neighbor_codes_num_ is always 1?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it can be further optimized by optimizer

@ShawnShawnYou ShawnShawnYou force-pushed the support-searcher branch 2 times, most recently from 8af23a1 to 6450d83 Compare January 20, 2025 15:53
@codecov Codecov
Copy link

codecov bot commented Jan 20, 2025 •
edited
Loading

Codecov Report

Attention: Patch coverage is 98.64865% with 1 line in your changes missing coverage. Please review.

@@            Coverage Diff             @@
##             main     #351      +/-   ##
==========================================
+ Coverage   90.94%   91.06%   +0.12%     
==========================================
  Files         133      134       +1     
  Lines        8459     8542      +83     
==========================================
+ Hits         7693     7779      +86     
+ Misses        766      763       -3
Flag Coverage Δ
cpp 91.06% <98.64%> (+0.12%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
common 95.75% <ø> (ø)
datacell 92.10% <100.00%> (+0.32%) ⬆️
index 91.01% <100.00%> (+<0.01%) ⬆️
simd 81.72% <ø> (+0.09%) ⬆️

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2b9ddac...9d96e8a. Read the comment docs.

LHT129
LHT129 reviewed Jan 21, 2025
View reviewed changes
src/impl/basic_searcher.cpp Outdated

void
BasicSearcher::SetGraphAndVector(GraphInterfacePtr graph_data_cell,
FlattenInterfacePtr vector_data_cell) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

split two function

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

LHT129
LHT129 reviewed Jan 21, 2025
View reviewed changes
src/impl/basic_searcher.cpp Outdated
std::pair<float, uint64_t>& current_node_pair,
Vector<InnerIdType>& to_be_visited_rid,
Vector<InnerIdType>& to_be_visited_id) const {
// to_be_visited_rid is used in redundant storage
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

redundant not used now

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added comment for distinguish to_be_visited_rid and to_be_visited_id in basic_search.h

LHT129
LHT129 reviewed Jan 21, 2025
View reviewed changes
src/impl/basic_searcher.cpp Outdated
}

void
BasicSearcher::SetGraphAndVector(GraphInterfacePtr graph_data_cell,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use interface name

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

LHT129
LHT129 reviewed Jan 21, 2025
View reviewed changes
src/impl/basic_searcher.cpp Outdated
uint32_t hops = 0;
uint32_t dist_cmp = 0;
uint32_t count_no_visited = 0;
Vector<InnerIdType> to_be_visited_rid(graph_data_cell_->MaximumDegree(), allocator_);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@ShawnShawnYou ShawnShawnYou added kind/feature New feature or request version/0.13 labels Jan 21, 2025
@ShawnShawnYou ShawnShawnYou self-assigned this Jan 21, 2025
inabao
inabao approved these changes Jan 21, 2025
View reviewed changes
Copy link
Collaborator

@inabao inabao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ShawnShawnYou ShawnShawnYou force-pushed the support-searcher branch 2 times, most recently from 9657fd4 to b789120 Compare January 23, 2025 08:39
inabao
inabao reviewed Jan 23, 2025
View reviewed changes
src/impl/basic_searcher.h Outdated
virtual MaxHeap
Search(GraphInterfacePtr graph_data_cell,
FlattenInterfacePtr vector_data_cell,
hnswlib::VisitedListPtr vl,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use ResourceObjectPool instead

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

src/impl/basic_searcher.h Outdated

virtual MaxHeap
Search(GraphInterfacePtr graph_data_cell,
FlattenInterfacePtr vector_data_cell,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

const GraphInterfacePtr& graph_data_cell
const FlattenInterfacePtr& vector_data_cell

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

LHT129
LHT129 reviewed Jan 23, 2025
View reviewed changes
src/impl/basic_searcher.cpp Outdated
uint32_t
BasicSearcher::visit(const GraphInterfacePtr& graph_data_cell,
std::shared_ptr<vsag::VisitedList> vl,
std::pair<float, uint64_t>& current_node_pair,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

vsag:: no useful

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

src/impl/basic_searcher.h Show resolved Hide resolved
src/impl/basic_searcher.h Show resolved Hide resolved
@ShawnShawnYou
support basic searcher
9d96e8a 
Signed-off-by: zhongxiaoyao.zxy <[email protected]>
wxyucs
wxyucs approved these changes Jan 23, 2025
View reviewed changes
Copy link
Collaborator

@wxyucs wxyucs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

LHT129
LHT129 approved these changes Jan 23, 2025
View reviewed changes
Copy link
Collaborator

@LHT129 LHT129 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ShawnShawnYou ShawnShawnYou merged commit 6ccab68 into main Jan 24, 2025
16 checks passed
@ShawnShawnYou ShawnShawnYou deleted the support-searcher branch January 24, 2025 03:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@inabao inabao inabao approved these changes

@LHT129 LHT129 LHT129 approved these changes

@wxyucs wxyucs wxyucs approved these changes

@jiaweizone jiaweizone Awaiting requested review from jiaweizone

Assignees

@ShawnShawnYou ShawnShawnYou

Labels
kind/feature New feature or request size/L version/0.13
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@ShawnShawnYou @wxyucs @inabao @LHT129