Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

<AnimatedImage> component #4175

Open
JonnyBurger opened this issue Aug 8, 2024 · 26 comments
Open

<AnimatedImage> component #4175

JonnyBurger opened this issue Aug 8, 2024 · 26 comments
Assignees
@Just-Moh-it
Labels
💎 Bounty

Comments

@JonnyBurger
Copy link
Member

TIL about ImageDecoder: https://developer.mozilla.org/en-US/docs/Web/API/ImageDecoder

Sample:

<!DOCTYPE html>
<head>
  <title>WebCodecs Animated GIF Renderer</title>
</head>
<canvas width="320" height="270"></canvas>
<br /><br />
<textarea style="width: 640px; height: 270px"></textarea>
<script>
  let imageDecoder = null;
  let imageIndex = 0;

  function log(str) {
    document.querySelector("textarea").value += str + "\n";
  }

  function renderImage(result) {
    const canvas = document.querySelector("canvas");
    const canvasContext = canvas.getContext("2d");

    canvasContext.drawImage(result.image, 0, 0);

    const track = imageDecoder.tracks.selectedTrack;

    // We check complete here since `frameCount` won't be stable until all data
    // has been received. This may cause us to receive a RangeError during the
    // decode() call below which needs to be handled.
    if (imageDecoder.complete) {
      if (track.frameCount == 1) return;

      if (imageIndex + 1 >= track.frameCount) imageIndex = 0;
    }

    // Decode the next frame ahead of display so it's ready in time.
    imageDecoder
      .decode({ frameIndex: ++imageIndex })
      .then((nextResult) =>
        setTimeout((_) => {
          renderImage(nextResult);
        }, result.image.duration / 1000.0)
      )
      .catch((e) => {
        // We can end up requesting an imageIndex past the end since we're using
        // a ReadableStrem from fetch(), when this happens just wrap around.
        if (e instanceof RangeError) {
          imageIndex = 0;
          imageDecoder.decode({ frameIndex: imageIndex }).then(renderImage);
        } else {
          throw e;
        }
      });
  }

  function logMetadata() {
    log("imageDecoder.type = " + imageDecoder.type);
    log("imageDecoder.tracks.length = " + imageDecoder.tracks.length);
    log("");

    function logTracks() {
      for (let i = 0; i < imageDecoder.tracks.length; ++i) {
        const track = imageDecoder.tracks[i];
        log(`track[${i}].frameCount = ` + track.frameCount);
        log(`track[${i}].repetitionCount = ` + track.repetitionCount);
        log(`track[${i}].animated = ` + track.animated);
        log(`track[${i}].selected = ` + track.selected);
      }
    }

    if (!imageDecoder.complete) {
      log("Partial metadata while still loading:");
      log("imageDecoder.complete = " + imageDecoder.complete);
      logTracks();
      log("");
    }

    imageDecoder.completed.then((_) => {
      log("Final metadata after all data received:");
      log("imageDecoder.complete = " + imageDecoder.complete);
      logTracks();
    });
  }

  function decodeImage(imageByteStream) {
    if (typeof ImageDecoder === "undefined") {
      log("Your browser does not support the WebCodecs ImageDecoder API.");
      return;
    }

    imageDecoder = new ImageDecoder({
      data: imageByteStream,
      type: "image/avif",
    });
    imageDecoder.tracks.ready.then(logMetadata);
    imageDecoder.decode({ frameIndex: imageIndex }).then(renderImage);
  }

  fetch("/2.avif").then((response) => decodeImage(response.body));
</script>

Crazy that it works with animated GIFs, AVIFs and animated WebPs!

We can re-implement the Gif component with it and also support other image formats!
@Just-Moh-it
Copy link
Contributor

hey hey hey… good weekend project and learning experience if you wanna assign!

@fitzmode
Copy link

fitzmode commented Aug 9, 2024

Hey @JonnyBurger would love to take a look at this. Thanks!

@JonnyBurger
Copy link
Member Author

Hey, I give it to @Just-Moh-it because he wrote first. Happy to give you the next one @fitzmode!

💎 This issue has a bounty on it!

Read our contributing guidelines:

/bounty 500

Criteria and hints

There should be a new component added to the core package (remotion) called <AnimatedImage>.

It should work similarly to the @remotion/gif package - fetch the file and parse the frames and display the current frame based on useCurrentFrame().

The @remotion/gif package has a separate implementation for development and rendering - this is not necessary.

When fetching a URL, the component should take a look at the Content-Type header. We will support three headers:

All work the same, just the type in ImageDecoder needs to be adjusted.
Other content types should be rejected.

There should be a good error message if ImageDecoder is not available (e.g. Firefox)

There should be a documentation page that aligns with other docs in the remotion package.

@algora-pbc Algora PBC
Copy link

algora-pbc bot commented Aug 9, 2024 •
edited
Loading

💎 $500 bounty • Remotion

Steps to solve:

  1. Get assigned: If you'd like to work on this issue, comment /attempt #4175 below to get assigned
  2. Submit work: Create a pull request including /claim #4175 in the PR body to claim the bounty
  3. Receive payment: 100% of the bounty is received 2-5 days post-reward. Make sure you are eligible for payouts

Thank you for contributing to remotion-dev/remotion!

Add a bounty • Share on socials

@hunxjunedo
Copy link
Contributor

Hey, would love to have a look at this, I'm in the row.

@akhilender-bongirwar
Copy link
Contributor

Hi, would like to work on this issue. I'm in the queue.

@amochuko
Copy link

Am also available to start working on this if chanced.

@amochuko
Copy link

/attempt #4175

@algora-pbc Algora PBC
Copy link

algora-pbc bot commented Aug 16, 2024

@amochuko: Another person is already attempting this issue. Please don't start working on this issue unless you were explicitly asked to do so.

@Mubashirshariq
Copy link

@JonnyBurger Can i get this one assigned ,i am really excited to get this component done

@Just-Moh-it
Copy link
Contributor

update: finishing this up, closing in!

@JonnyBurger
Copy link
Member Author

@Just-Moh-it can you open a Draft PR with the progress so far? maybe we can help finishing it

@zyronite
Copy link

Lemme give it a Trryyy!

@algora-pbc Algora PBC
Copy link

algora-pbc bot commented Sep 19, 2024

@j4nlksh: Another person is already attempting this issue. Please don't start working on this issue unless you were explicitly asked to do so.

@zyronite zyronite mentioned this issue Sep 19, 2024
Closed
3 tasks
@mrkirthi-24
Copy link
Contributor

@JonnyBurger lmk if this issue is up for grabs.

@onyedikachi-david
Copy link

@Just-Moh-it Are you still on this?

@arshad-muhammad
Copy link

@JonnyBurger since the issue still open I would like to contribute. Can you please assign this for me

@aadarsh-nagrath
Copy link

Whats status here ? Is this open for anyone to work ? @JonnyBurger

@karelnagel
Copy link
Contributor

karelnagel commented Nov 3, 2024 •
edited
Loading

I needed this, so I did create it on my own. Works with gif, webp and avif, if browser not supported or wrong content-type then falls back to use <Img /> instead.

Feel free to use it in remotion, or if still needed I could make this meet all the requirements and write the docs to solve this issue.

import { CSSProperties, useEffect, useMemo, useRef, useState } from 'react'
import { Img, continueRender, delayRender, useCurrentFrame, useVideoConfig } from 'remotion'

type ImageDecoderInit = {
  type: string
  data: ReadableStream | ArrayBuffer | ArrayBufferView | Blob | string | null
}

type ImageDecodeResult = {
  image: VideoFrame
  complete: boolean
}

type ImageDecoderVideoTrack = {
  selected: boolean
  frameCount: number
  repetitionCount: number
  frameSize: {
    width: number
    height: number
  }
}

type ImageDecoderTracks = {
  selectedIndex: number
  selectedTrack: ImageDecoderVideoTrack
}

declare class ImageDecoder {
  constructor(init: ImageDecoderInit)
  readonly tracks: ImageDecoderTracks
  readonly completed: Promise<void>
  decode(options?: { frameIndex?: number }): Promise<ImageDecodeResult>
  reset(): void
  close(): void
}

const ANIMATED_IMAGE_CONTENT_TYPES = ['image/gif', 'image/webp', 'image/avif']

type AnimatedImageMetadata = {
  width: number
  height: number
  fps: number | null
  frameCount: number
}

const useAnimatedImage = (src: string) => {
  const [handle] = useState(delayRender)
  const [decoder, setDecoder] = useState<ImageDecoder>()
  const [metadata, setMetadata] = useState<AnimatedImageMetadata>()

  useEffect(() => {
    const effect = async () => {
      if (typeof ImageDecoder === 'undefined') return console.error('ImageDecoder not available in this browser!')

      const response = await fetch(src)
      const contentType = response.headers.get('Content-Type')

      if (!contentType || !ANIMATED_IMAGE_CONTENT_TYPES.includes(contentType))
        return console.error(`Content type '${contentType}' is not supported!`)

      const decoder = new ImageDecoder({ data: response.body, type: contentType })
      await decoder.completed
      const frameCount = decoder.tracks.selectedTrack.frameCount

      // Decoding the first frame to get metadata
      const { image } = await decoder.decode({ frameIndex: 0 })
      const height = image.displayHeight
      const width = image.displayWidth
      // image.duration can be null if it's non-animated image
      const fps = image.duration ? 1000000 / image.duration : null

      setDecoder(decoder)
      setMetadata({ frameCount, height, width, fps })
    }
    effect().then(() => continueRender(handle))
  }, [src])

  return { decoder, metadata }
}

type AnimatedImageProps = {
  src: string
  style?: CSSProperties
  loopBehavior?: 'loop' | 'pause-after-finish'
  playbackRate?: number
}

export const AnimatedImage = ({ src, style, loopBehavior = 'loop', playbackRate = 1 }: AnimatedImageProps) => {
  const canvasRef = useRef<HTMLCanvasElement>(null)

  const { fps } = useVideoConfig()
  const frame = useCurrentFrame()

  const { decoder, metadata } = useAnimatedImage(src)

  const frameIndex = useMemo(() => {
    if (!metadata || !metadata.fps) return 0
    const currentFrame =
      loopBehavior === 'loop'
        ? (frame * playbackRate) % metadata.frameCount
        : Math.min(frame * playbackRate, metadata.frameCount)

    return Math.floor((currentFrame * metadata.fps) / fps)
  }, [frame, playbackRate, metadata, loopBehavior])

  useEffect(() => {
    const renderFrame = async () => {
      const canvas = canvasRef.current
      if (!decoder || !canvas) return

      const ctx = canvas.getContext('2d')
      if (!ctx) return

      const { image } = await decoder.decode({ frameIndex })
      ctx.drawImage(image, 0, 0)
    }

    const handle = delayRender()
    renderFrame().then(() => continueRender(handle))
  }, [decoder, frameIndex])

  if (!metadata) return <Img src={src} style={style} />
  return <canvas ref={canvasRef} width={metadata.width} height={metadata.height} style={style} />
}

@Harshit0741
Copy link

/attempt #4175

@algora-pbc Algora PBC
Copy link

algora-pbc bot commented Dec 1, 2024

@Harshit0741: Another person is already attempting this issue. Please don't start working on this issue unless you were explicitly asked to do so.

@Gyan-max
Copy link

/attempt #4175

@algora-pbc Algora PBC
Copy link

algora-pbc bot commented Dec 18, 2024

@Gyan-max: Another person is already attempting this issue. Please don't start working on this issue unless you were explicitly asked to do so.

@Gyan-max
Copy link

Hi @JonnyBurger, I’m really interested in contributing to this feature. If it’s possible to get assigned or collaborate on this, I’d love to help! If this issue is already assigned, I’d be happy to work on future issues. 😊

@JonnyBurger
Copy link
Member Author

@Gyan-max Sorry, we are already dealing with this!

Thanks for your interest though!

@FayadIdriss
Copy link

I can work on this ticket I can be assigned

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Just-Moh-it Just-Moh-it

Labels
💎 Bounty
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat: Add <AnimatedImage> for GIF, WebP, and AVIF Support! zyronite/remotion