[WIP][Quantization] Update default observer to be MSE

[shanjiaz]

Description
This PR addresses the following issue. Update such that the MSE is used as the default observer as opposed to MinMax.

Testing
Ran examples/quantization_w4a16 and inspected the observer. See QuantizationArgs:

weights=QuantizationArgs(num_bits=4, type='int', symmetric=True, group_size=128, strategy='group', block_structure=None, dynamic=False, actorder=None, observer='mse', observer_kwargs={})

More details:

Concern
@anmarques @eldarkurtic Wanted to reach out to confirm if this is what we wanted : )

[rahul-tuli]

Let's go!

[brian-dellabetta]

This seems like a pretty significant one-line change, that would affect pretty much every quantization pathway. Do we always want MSE over MinMax? Should we start a CHANGELOG so users are aware of internal changes like this?

[kylesayrs]

@brian-dellabetta : @dalistarh has validated that the MSE observer is better across the board (within reason). Logging changes is another topic outside of the scope of this PR I believe

[kylesayrs]

Since observers are only used by LLM Compressor, this change is safe to make without affecting existing checkpoints.

Just FYI for the future, this is not the case for actorder, as the checkpoints do not write the actorder explicitly, so modifying the CT checkpoint default will modify existing checkpoints. If we want to modify the actorder default, it will likely require adding an actorder field to GPTQModifier and resolving that with existing QuantizationArgs.

But side note aside, looks great!

[shanjiaz]

Since observers are only used by LLM Compressor, this change is safe to make without affecting existing checkpoints.

Just FYI for the future, this is not the case for actorder, as the checkpoints do not write the actorder explicitly, so modifying the CT checkpoint default will modify existing checkpoints. If we want to modify the actorder default, it will likely require adding an actorder field to GPTQModifier and resolving that with existing QuantizationArgs.

But side note aside, looks great!

Thanks Kyle! Will keep that in mind.

[dsikka]

This seems like a pretty significant one-line change, that would affect pretty much every quantization pathway. Do we always want MSE over MinMax? Should we start a CHANGELOG so users are aware of internal changes like this?

We should run through our lm-eval testing after this change lands. Ive already spoken to Helen about running the test

[dsikka]

We likely need to do two sets of validations before landing

1. Run through our lm-eval tests to ensure no regression
2. Validate timings/potential slowdowns.

We can connect on how to set this up @shanjiaz

[shanjiaz]

Test results available here.

[brian-dellabetta]

First PR! 🥳 🥇