Converting Quantised PyTorch Models #40
Replies: 3 comments 5 replies
-
|
Have you ever tried to writing your custom nobuco.converter? It might be much easier than you think. When I have time, I would also try a bit. Although I am not the owner of this project, I think your question is just valid for this project. Nobuco is well structured to convert from the fundamental torch functions to the entire model recursively. One person uses only small parts of torch, but we as a group use almost all the parts of torch. If each of us implements Unimplemented nodes for our own sake, Nobuco will be powered to convert most of torch models at some point. |
Beta Was this translation helpful? Give feedback.
-
|
I have written my own converters! Sorry maybe I wasn't clear in my post, I have written converters for Minimal code example: import torch
from torch import nn
from torch._ops import ops
import nobuco
from nobuco import ChannelOrder, ChannelOrderingStrategy
import tensorflow as tf
class Model(nn.Module):
def __init__(self, weight, bias):
super().__init__()
weight = torch.quantize_per_tensor_dynamic(weight, torch.qint8, False)
self.packed = torch.ops.quantized.linear_prepack(weight, bias)
def forward(self, x: torch.Tensor):
x = torch.quantize_per_tensor(x, 1.0, 128, torch.quint8)
x = ops.quantized.linear(x, self.packed, 1.0, 128)
x = x.dequantize()
return x
@nobuco.converter(torch.quantize_per_tensor)
def quantise(x: torch.Tensor, scale: float, zero: int, dtype: torch.dtype):
assert dtype == torch.quint8
min_q = -zero * scale
max_q = (255 - zero) * scale
def func(x: tf.Tensor, *args):
return tf.quantization.quantize(x, min_q, max_q, tf.quint8)[0]
return func
@nobuco.converter(torch.Tensor.dequantize)
def dequantise(x: torch.Tensor):
scale, zero = x.q_scale(), x.q_zero_point()
min_q = -zero * scale
max_q = (255 - zero) * scale
def func(x: tf.Tensor):
return tf.quantization.dequantize(x, min_q, max_q)
return func
@nobuco.converter(ops.quantized.linear)
def linear_quantized(x: torch.Tensor, packed, out_scale, out_zero):
weight, bias = ops.quantized.linear_unpack(packed)
def func(x: tf.Tensor):
# TODO: How to implement this?
pass
return func
def convert(model: nn.Module):
keras_model = nobuco.pytorch_to_keras(
model,
args=[torch.rand((1, 10))],
kwargs={},
inputs_channel_order=ChannelOrder.TENSORFLOW,
outputs_channel_order=ChannelOrder.TENSORFLOW,
)
return keras_model
model = Model(torch.rand((10, 10)), torch.rand((10,)))
converted = convert(model)Note all the errors about unsupported |
Beta Was this translation helpful? Give feedback.
-
I tried to add support for these dtypes lately, and that's a bigger task than I expected. For starters, quantized tensors in Tensorflow are not even tensors but Like you said, quantized ops are nowhere to be seen in Tensorflow.
Yes,
Dunno, my experience with that stuff is very limited.
I wouldn't have created Nobuco If I knew a better way to deploy models on the mobile and the web.
TFLite converter can perform post-training quantization automatically, why would do the same thing by hand?
I'd like to, I just don't have a good idea of how to approach that. Looks like there are lots of things to consider. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the answers! Good to know about the If the On doing PTQ with the TFLite converter, this is actually what I started with, but for some reason it absolutely destroyed output quality (the model in a GNN, which support for in general seems super flaky so it might just be a bug). Doing PTQ in PyTorch gives much better results (and also I have the option of doing QAT if I need even more accuracy). Finally, thanks for the pointer to |
Beta Was this translation helpful? Give feedback.
-
Yes, I need to keep them around. One reason is that
As I can see it, the
This mirrors my experience with vision models: complete garbage after full integer quantization. |
Beta Was this translation helpful? Give feedback.
-
|
My attempt at implementing dequantize was: @nobuco.converter(torch.Tensor.dequantize)
def dequantise(x: torch.Tensor):
scale, zero = x.q_scale(), x.q_zero_point()
min_q = -zero * scale
max_q = (255 - zero) * scale
def func(x: tf.Tensor):
return tf.quantization.dequantize(x, min_q, max_q)
return func(i.e. estimating the min and max from the torch tensor, and assuming those values don't change when it gets called in the Interesting to hear about Nice to know I'm not the only one getting terrible quality from PTQ in TensorFlow lol. |
Beta Was this translation helpful? Give feedback.
-
|
Alright, I think I'm onto something here. Got it to work for a simple quantized model. Try it: pip install https://github.com/AlexanderLutsenko/nobuco/archive/quantized.zipWhether the converted model will be properly quantized by TFLite is another question. |
Beta Was this translation helpful? Give feedback.
-
|
Woah nice! At least with the TFLite benchmark tool using XNNPack on x86, the fully connected layer in the exported TFLite model is still ran in float32, but just being able to export a model that produces correct results is still cool. Surrounding operators with dequantising and quantising again feels a lot like QDQ-style quantisation in ONNX, but yeah looks like somewhere either in the converter or interpreter/delegate it isn't recognising that we would like to run the fully connected layer in int8. |
Beta Was this translation helpful? Give feedback.
-
This is somewhere between a Nobuco feature request and just general questions about how quantisation in TensorFlow/TFLite works in case anybody who sees this knows.
Essentially my problem is that I have a quantised PyTorch model (manually quantised, so I am directly calling
torch.quantize_per_tensor,torch.ops.quantized.linear,torch.ops.quantized.layer_normetc...) and I want to convert this model to TFLite for deployment.For the non-quantised model, Nobuco does a great job (#36 was a bit funky and there were a couple operators with missing support but adding the
@nobuco.converters was surprisingly intuitive). For the quantised model, I am having a harder time.One problem is that when converting, I get a ton of errors about
quint8andqint8dtypes not being supported. I haven't yet dug into these error messages enough to figure out if this is a sign of a fundamental problem or just an obvious case not being handled somewhere, mainly because I've been more concerned by a second issue...Specifically, intercepting the quantised ops with
@nobuco.converters works like any other built-in PyTorch operators, but I am struggling to find TensorFlow functions to replace them with.torch.quantize_per_tensorbecomestf.quantization.quantizeandtorch.Tensor.dequantizebecomestf.quantization.dequantizebut beyond that, as far as I can tell, TensorFlow doesn't expose quantised versions of stuff like dense layers (though it does expose quantised concatenation... for some reason?). My best guess for why, looking at the APIs for TensorFlow quantisation, is that these operators don't really exist inside TensorFlow itself, only in TFLite.BUT of course it is possible to have models in TensorFlow that "act" like they are quantised (both during training in TensorFlow and when exporting to TFLite) - this is necessary for quantisation-aware training to work! The problem is that the APIs to create these seem really opaque. Instead of creating individual quantised layers and being able to set their
zero_pointandscaleyou just calltfmot.quantization.keras.quantize_modeland this does a bunch of stuff behind-the-scenes.So, context covered, specific questions time (to anyone who thinks they might know):
tfmot.quantization.keras.quantize_modelworks under-the-hood by inserting fake quantisation layers around ordinary TensorFlow operators (and then the TFLite exporter looks for these fake quantisation layers and replaces them with actual quantised operators), then I could try to do a similar thing inside the@nobuco.converters? Does this sound realistic/feasible?torch.export)Beta Was this translation helpful? Give feedback.
All reactions