LUT Sampling in React Native Skia Shader Returns Uniform Color

[Hello1nside]

Hello,

I'm working on applying LUT (Lookup Table) filters to images in a React Native Expo app using @shopify/react-native-skia. However, when I attempt to sample the LUT in my shader, I'm encountering an issue where the shader consistently returns a uniform color (either orange or white), instead of applying the LUT correctly across the image.

I've tried multiple approaches, but the result is still a solid color, even when attempting to directly sample from specific UV coordinates within the LUT. Below are some details of my attempts, followed by the shader code and the results I'm seeing.

Environment:
React Native: 0.74.2
React Native Skia: 1.2.3
Expo: ^51.0.14

Problem Description:

1. Sampling from the LUT using image color channels (color.rg) as UV coordinates returns a uniform orange color;
2. Directly sampling the center of the LUT with fixed UV coordinates (e.g., float2(0.5, 0.5)) also returns a uniform color (orange);
3. Flipping the UV coordinates and sampling the LUT results in a white color across the entire image;
4. When I directly render the LUT texture as a background (using pos as UV coordinates), it renders the LUT image correctly, which suggests that the LUT is being loaded, but something is going wrong when using color.rg values to sample from the LUT.

App.tsx:

import { Canvas, Fill, ImageShader, Shader } from "@shopify/react-native-skia";
import { Dimensions, StyleSheet, View } from "react-native";
import { source } from "./transitions";
import { useImage } from "@shopify/react-native-skia";

export default function App() {
  const { width, height } = Dimensions.get("window");
  const image = useImage(require('./assets/bugatti.jpg')); // image
  const lut = useImage(require('./assets/LUT.png')); // LUT

  if (!image || !lut) {
    return null;
  }

  return (
    <View style={styles.container}>
      <Canvas style={styles.container}>
        <Fill>
          <Shader source={source}>
            <ImageShader
              fit="cover"
              image={image}
              width={300}
              height={300}
            />

            <ImageShader
              fit="cover"
              image={lut}
              width={300}
              height={300}
            />
          </Shader>
        </Fill>
      </Canvas>
    </View>
  );
}

const styles = StyleSheet.create({
  container: {
    width: 300,
    height: 300
  },
});

transition.ts:

import { Skia } from "@shopify/react-native-skia";

export const source = Skia.RuntimeEffect.Make(`
  uniform shader image;
  uniform shader lut;

  half4 applyLUT(half4 color) {
    // Define the grid size and tile size
    half2 tiles = half2(8.0, 8.0);
    half2 tileSize = half2(64.0, 64.0);

    // Normalize the input color (0.0 to 1.0 range)
    half3 normalizedColor = color.rgb;

    // Determine the index of the tile based on the blue channel
    half index = normalizedColor.b * (tiles.x * tiles.y - 1.0);

    // Calculate the tile index within the 8x8 grid
    half2 tileIndex;
    tileIndex.y = floor(index / tiles.x);
    tileIndex.x = floor(index - tileIndex.y * tiles.x);

    // Map the red and green channels to the UV coordinates within the tile
    // half2 tileUV = mix(0.5 / tileSize, (tileSize - 0.5) / tileSize, normalizedColor.rg);
    half2 tileUV = normalizedColor.rg * (tileSize - 1.0) / tileSize;

    // Calculate the final texture coordinates for the LUT sampling
    // half2 tableUV = tileIndex / tiles + tileUV / tiles;
    half2 tableUV = (tileIndex + tileUV) / tiles;

    // Sample the LUT texture at the computed coordinates
    half3 lookUpColor = lut.eval(tableUV).rgb;

    // Interpolate between two tiles for smoother transitions
    half index_min = min(62.0, floor(index));
    half index_max = index_min + 1.0;

    half2 tileIndex_min;
    tileIndex_min.y = floor(index_min / tiles.x);
    tileIndex_min.x = floor(index_min - tileIndex_min.y * tiles.x);

    half2 tileIndex_max;
    tileIndex_max.y = floor(index_max / tiles.x);
    tileIndex_max.x = floor(index_max - tileIndex_max.y * tiles.x);

    half2 tableUV_1 = tileIndex_min / tiles + tileUV / tiles;
    half2 tableUV_2 = tileIndex_max / tiles + tileUV / tiles;

    half3 lookUpColor_1 = lut.eval(tableUV_1).rgb;
    half3 lookUpColor_2 = lut.eval(tableUV_2).rgb;

    half3 finalColor = mix(lookUpColor_1, lookUpColor_2, index - index_min);

    // Return the new color with the same alpha as the input
    return half4(finalColor, color.a);
  }

  half4 main(float2 pos) {
    // Sample the input image at the current position
    half4 inputColor = image.eval(pos);

    // Apply the LUT to the input color
    half4 outputColor = applyLUT(inputColor);

    return outputColor;
  }`)!;

if (!source) {
  throw new Error("Couldn't compile the shader")
}

Tests and Results:

1. Simple shader code, that also returns orange color

half4 applyLUT(half4 color) {
  // Use the red and green channels as UV coordinates
  float2 uv = color.rg;
  // Sample the LUT using the UV coordinates
  half4 lutColor = lut.eval(uv);
  return lutColor;
}

half4 main(float2 pos) {
  // Sample the input image at the current position
  half4 inputColor = image.eval(pos);
  // Apply the LUT using the image color channels
  half4 outputColor = applyLUT(inputColor);
  return outputColor;
}

5. Sampling from fixed coordinates in the LUT:

float2 uv = float2(0.5, 0.5); // Center of the LUT
half4 lutColor = lut.eval(uv);
return lutColor;

Result: Uniform orange color.

6. Flipping UV coordinates:

float2 uv = float2(1.0 - pos.x, 1.0 - pos.y); // Flipped coordinates
half4 lutColor = lut.eval(uv);
return lutColor;

Result: Uniform white color.

7. Direct LUT Rendering:

half4 main(float2 pos) {
  half4 lutColor = lut.eval(pos); // Directly sample LUT using pos
  return lutColor; // Display the LUT image
}

Result: The LUT image is rendered correctly on the screen, showing that the LUT is being loaded, but it's not properly sampled when used with color.rg as UV coordinates.

What I've Tried:

• Normalizing the color.rg values to ensure they are in the 0-1 range.
• Clamping the UV coordinates before sampling the LUT.
• Testing multiple LUT files, all of which result in the same issue (uniform color).

Expected Behavior:
I expect the LUT to apply color transformations to the image based on its RGB values. Instead of a uniform color, the LUT should modify the image's color channels correctly, based on the LUT's contents.

Actual Behavior:
The shader returns a uniform color (orange or white for this LUT, for other LUTs there is other colors), and the LUT doesn't seem to be sampled dynamically across the texture.

Conclusion:
It seems that the LUT texture is not being sampled properly based on the image's color.rg values. I would appreciate any insights into what might be causing this issue, or if there are any specific steps I can take to debug the LUT sampling.

Attached Files:
I have also attached the image and LUT files I am using in my project for further reference and testing:

• The image I'm applying the LUT to:
  
• The LUT file being used in the shader:
  
• Another LUT that I used as well (it returns red color):
  
• Screenshot from the simulator phone showing the issue where the LUT sampling results in a uniform orange color:

Please feel free to use these files to replicate the issue.

Any insights or solutions are greatly appreciated!

Thank you!

[Hello1nside]

I found a solution in the discussion: #1436
Issue solved.