[BUG] Frequent dropped frames

[zkelly1]

Please only report one bug per issue!

Describe the bug
We are witnessing frequent large, frame drops from the camera, even with the maximum buffer size (150 frames).

To Reproduce
'''python

from picamera2 import Picamera2

# Initialize camera 
cam: Picamera2 = Picamera2()

# Select the mode to put the sensor in
# (ie, mode for high fps/low res, more pixel HDR etc)
sensor_mode: dict = cam.sensor_modes[4] # 4		

# Set the mode
cam.configure(cam.create_video_configuration(sensor={'output_size':sensor_mode['size'], 'bit_depth':sensor_mode['bit_depth']}, 
                                             main={'size':sensor_mode['size']}, 
                                             raw=sensor_mode,
                                             queue=True,
                                             buffer_count=150))

# Ensure the frame rate; This is calculated by
# FPS = 1,000,000 / FrameDurationLimits 
# e.g. 206.65 = 1000000/FDL => FDL = 1000000/206.65
# 200 = 
frame_duration_limit = int(np.ceil(1000000/CAM_FPS))
cam.video_configuration.controls['NoiseReductionMode'] = 0
cam.video_configuration.controls['FrameDurationLimits'] = (frame_duration_limit,frame_duration_limit) # *2 for lower,upper bound equal

# Set runtime camera information, such as auto-gain
# auto exposure, white point balance, etc
# Note, AeEnable changes both AEC and AGC		
cam.video_configuration.controls['AwbEnable'] = 0
cam.video_configuration.controls['AeEnable'] = 0  

#cam.video_configuration.controls['AnalogueGain'] = 2
#cam.video_configuration.controls['ExposureTime'] = initial_exposure


# HARDCODED FOR THE TEST
cam.video_configuration.controls['AnalogueGain'] = 1
cam.video_configuration.controls['ExposureTime'] = 1300

from picamera2 import Picamera2

# Connect to and set up camera
print(f"Initializing camera")
cam: Picamera2 = initialize_camera(initial_gain, initial_exposure)
gain_change_interval: float = 0.250 # the time between AGC adjustments 

# Begin Recording and capture initial metadata 
cam.start("video")  
initial_metadata: dict = cam.capture_metadata()
current_gain, current_exposure = initial_metadata['AnalogueGain'], initial_metadata['ExposureTime']

# HARDCODE FOR A SPECIFIC TEST 
#current_gain, current_exposure = 10, 4839

# Make absolutely certain Ae and AWB are off 
# (had to put this here at some point) for it to work 
cam.set_controls({'AeEnable':0, 'AwbEnable':0})   

 # Initialize a contiguous memory buffer to store 1 second of frames 
# + settings in this is so when we send them to be written, numpy does not have 
# to reallocate for contiguous memory, thus slowing down capture
frame_buffer: np.array = np.zeros((CAM_FPS, 480, 640), dtype=np.uint8)
settings_buffer: np.array = np.zeros((CAM_FPS, 2), dtype=np.float32)
frame_timings_buffer: np.array = np.zeros((CAM_FPS,2), dtype=float) 
#cpu_info_buffer: np.array = np.zeros((CAM_FPS,2), dtype=float) 


# Initialize the last time we changed the gain as the current time
last_gain_change: float = time.time()  

# Capture indefinite frames
frame_num: int = 0 
while(not stop_flag.is_set()):
    #frame_num_mod: int = 

    # Capture the current time
    #current_time: float = time.time()
    
    # Capture the frame and splice only the odd cols (even cols have junk content)
    frame: np.array = cam.capture_array('raw')[:, 1::2]

    # Record when this capture process completed
    #finish_capture_time: float = time.time()

    # Capture information about the state of the machine 
    #cpu_usage = psutil.cpu_percent(interval=0)
    #cpu_clockspeed = psutil.cpu_freq().current
    #cpu_temp = psutil.sensors_temperatures()['cpu_thermal'][0].current
    #memory_usage = psutil.virtual_memory().percent

    # Store the frame + settings into the allocated memory buffers
    frame_buffer[frame_num % CAM_FPS] = frame
    #settings_buffer[frame_num % CAM_FPS] = [current_gain, current_exposure]
    #frame_timings_buffer[frame_num % CAM_FPS] = [current_time, finish_capture_time]
    #cpu_info_buffer[frame_num % CAM_FPS] = [cpu_usage, cpu_clockspeed]

    # Change gain every N ms
    """
    if((current_time - last_gain_change) > gain_change_interval):
        # Take the mean intensity of the frame
        mean_intensity = np.mean(frame, axis=(0,1))
        
        # Feed the settings into the the AGC 
        ret = AGC(mean_intensity, current_gain, current_exposure, 0.95, AGC_lib)

        # Retrieve and set the new gain and exposure from our custom AGC
        #new_gain, new_exposure = ret['adjusted_gain'], int(ret['adjusted_exposure'])

        # HARD CODE FOR A SPECIFIC TEST 
        new_gain, new_exposure = 1, 1300

        cam.set_controls({'AnalogueGain': new_gain, 'ExposureTime': new_exposure}) 
        
        # Update the current_gain and current_exposure, 
        # wait for next gain change time
        last_gain_change = current_time
        current_gain, current_exposure = new_gain, new_exposure
    """

    # Record the next frame number
    frame_num += 1 

    # If we have now captured one second worth of frames, send the frame buffer 
    # to be written 
    if(frame_num % CAM_FPS == 0):
        #write_queue.put((frame_buffer, frame_num, settings_buffer, frame_timings_buffer, cpu_info_buffer))
        write_queue.put((frame_buffer, frame_num, settings_buffer, frame_timings_buffer))

'''

Expected behaviour
The camera captures 200 FPS with minimal frame loss. Any potential dropped frames are filled in by the 3/4th of a second buffer OR frame drops can be caught by elapsed time.

Console Output, Screenshots

Above is an image showing several dropped frames when the camera is observing a 5hz sinusoidal modulation. These are relatively frequent.

Additionally, this tends to coincide with a spike in CPU usage to 100% as well as a decrease in clockspeed. It also tends to correlated with spikes in elapsed time of the capture_array function. Though, this is an imperfect measurement, as sometimes there are equally large spikes in the elapsed time without any dropped frames.

Hardware :
We are running a Raspberry Pi 5 using a Sony IMX 219 camera connected via HDMI.

Additional context
Add any other context about the problem here.

[davidplowman]

Hi, and thanks for the question.

I might try and start with a really simple example, with no other processing, in order to see if you are getting anything close to the framerate you want. Maybe like this:

import time
from picamera2 import Picamera2

cam = Picamera2()
config = picam2.create_preview_configuration(raw=cam.sensor_modes[4], controls={'FrameRate': 200}, buffer_count=32)
cam.configure(config)
cam.start()
cam.capture_metadata()  # first frame takes a while, so don't count it

start = time.now()
count = 0
while time.now() - start < 10:
    cam.capture_metadata()
    count += 1

print("Achieved", count / 10, "fps")

I've not tried that, so there might be typos, but you get the idea.

Having said that, I wouldn't like to promise that 200fps is achieveable in Python code, , especially once you start adding in more processing, but something like the code above would enable you to get an idea.