[luci/pass] Experiment with decomposed LayerNorm to InstanceNorm

[seanshpark]

From #14447 (comment)

Lets' do some performance experiments with

input:

Input [N, L, D] -> (decomposed) LayerNorm [N, L, D]

output

Input [N, L, D] -> Transpose [N, D, L] -> Reshape [N, 1, D, L] ->
InstanceNorm [N, 1, D, L] -> Reshape [N, D, L] -> Transpose [N, L, D]

with our NPU backend

What to do (with draft)

• choose some models for experiments
• introduce CircleLayerNorm virtual IR
• add pass to fuse to CircleLayerNorm
• add pass to convert to CircleInstanceNorm with Transpose + Reshape
• add tests for transformation
• measure in NPU with source model vs converted model

[seanshpark]

pytorch source

import onnx
import torch
import torch.nn as nn


class LayerNormNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.ln = nn.LayerNorm((4))

    def forward(self, x):
        out = self.ln(x)
        return out


net = LayerNormNet()
inp = torch.randn(1, 16, 4)

torch.onnx.export(net, inp, "ln11.onnx", opset_version=11)
onnx.shape_inference.infer_shapes_path('ln11.onnx', 'ln11-si.onnx')

torch.onnx.export(net, inp, "ln17.onnx", opset_version=17)
onnx.shape_inference.infer_shapes_path('ln17.onnx', 'ln17-si.onnx')

use ln11-si.onnx

[seanshpark]

circle model

before	after

[seanshpark]

Instance norm with 2 x Transpose + 2 x Reshape uses more cycles than decomposed LayerNorm.
Doesn't look worth adding this transformation.

ref internal ONE/issues/127