Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CPU] Change kvcache default type of PagedAttention to u8 for CPU plugin #1206

Open
wants to merge 8 commits into
base: master
Choose a base branch
from
+65 −75
Open

[CPU] Change kvcache default type of PagedAttention to u8 for CPU plugin #1206

Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
ac18dd4
change kvcache default type to u8 for cpu plugin
luo-cheng2021 Nov 13, 2024
ffef13e
use f32 for hint: EXECUTION_MODE_HINT:ACCURACY
luo-cheng2021 Nov 14, 2024
b9a05f0
Merge remote-tracking branch 'upstream/master' into luocheng/pa_kv_de…
luo-cheng2021 Nov 14, 2024
9efab9d
Revert "[GHA]: hardcode OpenVINO commit (#1212)"
luo-cheng2021 Nov 14, 2024
eb44c6e
Merge remote-tracking branch 'upstream/master' into luocheng/pa_kv_de…
luo-cheng2021 Dec 31, 2024
400e41c
fix ci errors
luo-cheng2021 Dec 31, 2024
9de7c89
Merge branch 'master' into luocheng/pa_kv_default_u8
ilya-lavrenov Jan 4, 2025
07f0382
Merge branch 'master' into luocheng/pa_kv_default_u8
ilya-lavrenov Jan 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 3 additions & 19 deletions src/cpp/src/device_config.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,30 +36,14 @@ class DeviceConfig {
m_block_size = get_block_size_by_device(device);

if (m_device == "CPU") {
auto inference_precision = core.get_property(device, ov::hint::inference_precision);
m_kv_cache_type = inference_precision == ov::element::bf16 ? ov::element::bf16 : ov::element::f16;

// if user sets precision hint, kv cache type should be changed
const auto inference_precision_it = plugin_config.find(ov::hint::inference_precision.name());
if (inference_precision_it != plugin_config.end()) {
const auto inference_precision = inference_precision_it->second.as<ov::element::Type>();
if (inference_precision == ov::element::f32) {
m_kv_cache_type = ov::element::f32;
} else if (inference_precision == ov::element::f16) {
m_kv_cache_type = ov::element::f16;
} else if (inference_precision == ov::element::bf16) {
m_kv_cache_type = ov::element::bf16;
} else {
// use default f32
m_kv_cache_type = ov::element::f32;
}
}

// if user sets ov::kv_cache_precision hint
const auto kv_cache_precision_it = plugin_config.find(ov::hint::kv_cache_precision.name());
if (kv_cache_precision_it != plugin_config.end()) {
const auto kv_cache_precision = kv_cache_precision_it->second.as<ov::element::Type>();
m_kv_cache_type = kv_cache_precision;
} else {
// x86 and arm have different default kv cache type
ilya-lavrenov marked this conversation as resolved.
Show resolved Hide resolved
m_kv_cache_type = core.get_property(device, ov::hint::kv_cache_precision);
}
} else if (m_device.find("GPU") != std::string::npos) {
auto inference_precision = core.get_property(device, ov::hint::inference_precision);
Expand Down
Loading