avatar_interaction

The repository to hold all interaction related code for avatar

TODO AFTER FINAL

Minors

• snd device string auto search
• documentation
• modular design

Majors

• NDI HX implementation to reduce bandwidth
• WebRTC implementation
• Watchdog

TODO

Avatar Audio Video Processing

• Setup the new router

• NDI audio processing

  • Understand how snd_pcm_readi() works
  • Understand period, frame, buffer, sample rate, bit depth
  • Create a local capture and playback program
  • Send out NDI packet
  • Receive NDI packet
  • Solve underrun issue (solved by adding a delay on purpose)
  • Try capturing device with MMAP mode
  • Look into frame drops along the time. (Seems related to how I wrote the receiver)
  • Look into receiver issue. The speaker and headphone device name not shown in alsa.
  • Resource busy (Alsa)
    • Run fuser -fv /dev/snd* to show what's using the port, most of the time it will be pulseaudio
    • Run pulseaudio --kill to turn off pulseaudio
    • Or use 'ndi_recv_pa' to use pulseaudio
  • Mixer
  • AEC implementation with pulseaudio (Solved with a RMS based echo cancellation)

• NDI video processing

  • MJPEG format support (SO HARD!!!!) Switch to FFMPEG might be the option
  • Send audio along with video
    • Tune the synchronization
  • How to generate 60fps video?? Maybe play with the buffer too? (solved with better router)
  • YUYV format support
  • Crop
    • SDL render issue with cropped video
  • Resolution
  • Solve memory leak in MJPEG mode
  • Test with discovery server
  • Try with multiple NICs

• AV integration

  • Integrate 3 mics, 3 cameras, 1 speaker on robot side
  • Integrate 1 camera, 1 headphone on operator side
  • assess audio video desync

• UI intergration

  • Template created with font color/size/orientation/background color support
  • ROS integration

• ffmpeg for video streaming

  • Tunning ffmpeg for different streaming options: resolution, ratio, codec, fps, stream protocol

• Try Ultragrid (The Ultragrid would crash the system somehow)

  • Use Ultragrid to transmit Jabra and Insta with compression.

Sender: ffmpeg -f v4l2 -framerate 60 -video_size 1920x1080 -input_format mjpeg -i /dev/video4 -preset faster -pix_fmt yuv420p -f mpegts -flush_packets 0 udp://127.0.0.1:23000

Sender(GPU): ffmpeg -f v4l2 -framerate 60 -video_size 1920x1080 -input_format mjpeg -hwaccel cuda -i /dev/video2 -f mpegts -flush_packets 0 udp://127.0.0.1:23000

Receiver: ffplay udp://127.0.0.1:23000 -fflags nobuffer ffplay udp://127.0.0.1:23000 -fflags nobuffer -framedrop -flags low_delay

Helpful links: https://trac.ffmpeg.org/wiki/StreamingGuide#Latency https://stackoverflow.com/questions/47292785/recording-from-webcam-using-ffmpeg-at-high-framerate

Inspired from https://github.com/marcosps/v4l2_webcam