Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BugFix] Stop silent failures on compressed-tensors parsing #9381

Merged
merged 4 commits into from
Oct 18, 2024
+23 −13
Merged

[BugFix] Stop silent failures on compressed-tensors parsing #9381

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
a6b6121
stop silently failing when input_act args are incorrect
dsikka Oct 15, 2024
26b4f1e
bump version
dsikka Oct 16, 2024
bbacbbd
update version; clean-up condition
dsikka Oct 16, 2024
c19ad75
Delete vllm/commit_id.py
dsikka Oct 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion requirements-common.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,4 +31,4 @@ pyyaml
six>=1.16.0; python_version > '3.11' # transitive dependency of pandas that needs to be the latest version for python 3.12
setuptools>=74.1.1; python_version > '3.11' # Setuptools is used by triton, we need to ensure a modern version is installed for 3.12+ so that it does not try to import distutils, which was removed in 3.12
einops # Required for Qwen2-VL.
compressed-tensors == 0.6.0 # required for compressed-tensors
compressed-tensors == 0.7.0 # required for compressed-tensors
10 changes: 5 additions & 5 deletions vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,11 +100,12 @@ def from_config(cls, config: Dict[str, Any]) -> "CompressedTensorsConfig":
target_scheme_map[target][
"weights"] = QuantizationArgs.parse_obj(
quant_config.get("weights"))
try:

if is_activation_quantization_format(quant_format):
target_scheme_map[target][
"input_activations"] = QuantizationArgs.parse_obj(
quant_config.get("input_activations"))
except Exception:
mgoin marked this conversation as resolved.
Show resolved Hide resolved
else:
target_scheme_map[target]["input_activations"] = None

return cls(target_scheme_map=target_scheme_map,
Expand Down Expand Up @@ -244,9 +245,8 @@ def _get_scheme_from_parts(
group_size=weight_quant.group_size,
actorder=weight_quant.actorder)

# Detect If Activation Quantization.
# TODO @dsikka: clean-up conditions
if is_activation_quantization_format(self.quant_format):
# Will only be not None if is_activation_quantization_format is True
if input_quant:
if self._is_fp8_w8a8(weight_quant, input_quant):
is_fp8_w8a8_supported = self._check_scheme_supported(
CompressedTensorsW8A8Fp8.get_min_capability(), error=False)
Expand Down