Is the image square-resized before inputting in the NN ?

[ghost]

I dont see images "squared-and-resized" in the source code, but it seems resizing would normally be required. I wanted to ask whether the input image is resized before feeding it to the NN or not ?

[vladmandic]

Yes, images are automatically squared and resized using different resolutions for different models included in Human.

[vladmandic]

Yes, Human does padding to square images.
And you're right on slight differences of resising algorithms, but in real world it's not really visible.

[vladmandic]

a bit more details...

i do have tensor-based padding code, but by default human uses temporary canvas and any input is drawn on it before data is extracted. reason is that for for majority media inputs (video stream, etc.), it goes through interim canvas anyhow and this allows me to perform additional gl filters on it without any computational cost (flip, brightness, contrast, saturation, etc.).

it also allows me to work with pretty much any input type:

export type AnyCanvas = HTMLCanvasElement | OffscreenCanvas;
export type AnyImage = HTMLImageElement | typeof Image
export type AnyVideo = HTMLMediaElement | HTMLVideoElement
export type ImageObjects = ImageData | ImageBitmap
export type ExternalCanvas = typeof env.Canvas;
export type Input = Tensor | AnyCanvas | AnyImage | AnyVideo | ImageObjects | ExternalCanvas;

only if user provides tensor as input, then its resized and used-as-is without converting.
in that case, padding is performed only for specific models which are aspect-ratio sensitive. for example, no need to perform padding when extracting face box since its not aspect ratio sensitive and it will be squared after detection anyhow (before passing to mesh or emotion or age/gender).

but on the other hand, for body detection, it does explicit padding.

[ghost]

Interesting, I was actually writing again and am happy there was something related to read.

I don't understand well the canvas very well ( I guess you mean to clean up the image using some utilities from Canvas API ?). I dont get also well the TF-backend selection part just yet, but I was just wondering whether a resizing that does not keep aspect ratio (AR) would worsen the detection. I tested with this little code (I am only learning):

function padder1d(i: tf.Tensor3D, dType: 'int32' | 'float32' = 'int32') {
  const width = i.shape[0];
  const height = i.shape[1];
  if (width > height) {
    const pad = tf.zeros([width, width - height, 3], dType);
    return tf.concat([i, pad], 1);
  } else {
    const pad = tf.zeros([height - width, height, 3], dType);
    return tf.concat([i, pad], 0);
  }
}

As you say, it didn't make a difference compared to a resizing that would change the AR. However, I do not think this is technically correct (may effect may be negligible.); CNNs detect edges as far as I know, and the more you stretch with a bilinear or nk method, the more you blur edges.

(From the start it was actually unexpected to me that TF models aren't for any size, but I guess this isn't simple to do.)

[vladmandic]

I don't understand well the canvas very well

human creates one instance of temp canvas and any input that is not a tensor is drawn onto that canvas.
from there, i can use canvas gl context to use manually written glsl functions for image manipulation which is much faster than using canvas api. equally, canvas pixel data is then transfered to tensor using glsl code which is much faster than reading imagedata or anything already existing in canvas api. none of this is needed, it was my choice to do it this way as i can have a single universal preprocessor for all input types.

I was just wondering whether a resizing that does not keep aspect ratio (AR) would worsen the detection

it does, but some models are less sensitive to it up to a point where its not-an-issue. and some models are very sensitive to it.
it all depends how convolutions are performed.

From the start it was actually unexpected to me that TF models aren't for any size

they can be and some are, but to have it "for any resolution" requires that input is resized to a multiple of window and then processed in multiple loops, each one with different scale (basically a simplified striding). there is no such thing as "universal model" that works on any resolution, the question is how much pre-processing is done in the model vs in the code. and yes, i've found that most torch models have more preprocessing built-in which almost no tfjs models do so - its all about latency and making concicous decisions.

[ghost]

I see. Interesting explanations. I'll take a look to Canvas and glsl to understand more.