Skip to content

Module Guidelines

Jump to bottom
Mehmet Agaoglu edited this page May 12, 2020 · 9 revisions

The goal of this document is to describe the requirements of adding a new module to ReVAS. It includes a step-by-step instructions for creating a new module following these guidelines. If a new module is contributed to ReVAS without conforming to these guidelines, it will not be merged to master branch (pull request will be rejected).

Table of Contents

Input arguments

Each module in ReVAS must have two input arguments. The first one (also referred to as primary input) must be a full path to a file (video, or .mat file) or an input array (3D array for videos, 2D array for position, etc.). The second argument (secondary input) must be a params structure, which defines the parameters to be used in this particular module. The params structure can have any number or type of fields, however it must have at least an overwrite field, which will determine whether or not the output of this module will overwrite an existing file with the same name.

Depending on the type of operation that our new module will do, there might be additional restrictions or some required fields. For instance, if our new module is going to operate on videos, a badFrames field (which is a logical array indicating which frames are "bad" due to blinks or subject movement, etc.) is also required to propagate that info downstream in the processing pipeline, even if it is not used in our new module.

If we want to use the ReVAS GUI to show some progress or visualize the output of our module, we must add a field called enableVerbosity to params structure, along with the axesHandles field, indicating which axes object(s) will be used to do the visualization.

Output arguments

Each ReVAS module must have at least two output arguments. The first output argument (also referred to as primary output) is used as an input to the next module in the pipeline, therefore, a decision has to be made here. If the immediate output of our module is going to be fed into the next module in the pipeline, then this output variable (i.e., processed video, extracted motion, filtered motion, etc.) can be passed to the first output argument. If this module only acts like an intermediate processing step (e.g., creating a reference frame but passing the input video to the next module untouched), then first output argument must be the same as the first input argument and the primary product of the current module must be passed onto subsequent modules via the params structure.

The second output argument must be the params structure. This is (1) to enable chaining multiple modules back to back, (2) passing output of a bypass module to later stages in the pipeline.

Finally, a module might have three or more output arguments. MATLAB's varargout must be used for modules with more than two output arguments.

Unit testing

Each module in ReVAS must have a tester function. These functions must be located in testing/ folder and must have a name with the following format, Tester_<module-name>.m. Each tester should have at least two different scenarios (one with file input and another with array input) but more tests are strongly encouraged. Demo videos under demo/ folder can be used in a tester. The output of a tester function must be a logical flag (true/false). For examples and style guidelines or inspirations, check out the available testers.

See Writing a unit tester to see more detailed instructions on how to write a tester for your module.

Module types

Conceptually, there are three types of modules.

Core modules are the ones which take in a video array or eye motion traces or a file path to one of these, perform some processing steps on this input, and return a new output (in the form of a modified video, or processed motion traces). For instance, BandpassFilter.m takes in a video and applies a bandpass filter to each frame, and returns another video with these filtered frames.

Bypass modules also take one of the aforementioned inputs, however, they pass the input to the output as is. Bypass modules are used to create intermediate or auxiliary parameters or dependencies for subsequent modules in the pipeline. For instance, FindBlinkFrames.m module takes in a video, detects bad frames during which video quality drops or there is a blink, passes the input video as is to its first output argument and an array of labels badFrames as a field of its second output argument params (or writes badFrames to a .mat file if the first input is a full path to a video). Therefore, FindBlinkFrames.m can be considered as a Bypass module.

Hybrid modules can be considered as a combination of the previous two types. StripAnalysis.m module, for instance, takes in a video and returns extracted eye motion traces in its primary output. In addition, it returns same and additional information (e.g., position, timeSec, peakValueArray, etc.) as new fields under params structure. Another example is RemoveStimuli.m, which operates on videos, removes stimuli and returns a modified video array (or file) along with stimulus locations (and some other information) as additional output arguments (or file).

The following table classifies existing ReVAS modules into these different types.

Core Bypass Hybrid
  • TrimVideo.m
  • GammaCorrect.m
  • BandpassFilter.m
  • ReReference.m
  • FilterEyePosition.m
  • Pixel2Degree.m
  • Degree2Pixel.m
  • FindBlinkFrames.m
  • MakeReference.m
  • FindSaccadesAndDrifts.m
  • RemoveStimuli.m
  • StripAnalysis.m

A step-by-step guide to creating a new module

We want to create a new Hybrid module. Let us assume that this module takes in a video, inverts pixel values of each frame (just for the sake of an example), and randomly selects one of its frames and returns that frame and its frame number as additional outputs. If the first input is a path to a video, first output is also a path but to the processed video. In this case, additional outputs are written to a .mat file. If the first input is a 3D video array, the first output is also a 3D array containing the processed video. In this scenario, the randomly selected frame is returned as a field in params and its frame number is returned as the third output argument via varargout. We will call this new module DummyModule.m. Conceptually, it is going to be a Hybrid module.

Function declaration and header

ReVAS modules has to follow the input/output argument guidelines mentioned above. In addition, all modules must have a function header with the following style.

  • Function name with all input and output arguments.
  • A brief description of what the module does.
  • Input/output parameters and their meanings, types, default values, constraints.
  • Example usage (optional)
function [outputVideo, params, varargout] = DummyModule(inputVideo, params)
%[outputVideo, params, varargout] = DummyModule(inputVideo, params)
%
%   This is a dummy module, used to demonstrate how to add a new module to
%   ReVAS. Please see the wiki page at
%   https://github.com/lowvisionresearch/ReVAS/wiki/Module-Guidelines. It
%   takes a video (or a path to it) as first input, inverts pixel values of
%   the video and returns the modified video (or its new path) as output.
%
%   -----------------------------------
%   Input
%   -----------------------------------
%   |inputVideo| is the path to the video or a matrix representation of the
%   video that is already loaded into memory.
%
%   |params| is a struct as specified below.
%
%   -----------------------------------
%   Fields of the |params| 
%   -----------------------------------
%
%   overwrite         : set to true to overwrite existing files. Set to 
%                       false to params.abort the function call if the files
%                       already exist. (default false)
%   enableVerbosity   : set to true to report back plots during execution.(
%                       default false)
%   badFrames         : vector containing the frame numbers of the blink 
%                       frames. (default [])
%   axesHandles       : axes handle for giving feedback. if not provided, 
%                       new figures are created. (relevant only when
%                       enableVerbosity is true)
%
%   -----------------------------------
%   Output
%   -----------------------------------
%   |outputVideo| is the processed video (or full path to it).
%
%   |params| structure, including a new field called `randomFrame`.
%
%   |varargout| is a variable output argument holder.
%   varargout{1} = randomFrameNumber, the frame number of the randomly 
%   selected video array
%

Determine input type

All modules in ReVAS determine whether the output will be written to a new file by checking the variable type of its first input argument. If it is a character array, it is likely a full path to the file to be processed, and the output of this module will be written to a new file (with current module suffix appended to file name). In this case, the first output argument will be the full path the new file written by this module.

If the first input is an array (3D array for video input, 2D for eye motion traces), the output will not be written to a file but instead returned as first output argument. In this example, inputVideo is our primary input. The following block of code checks its type and sets a flag. If first input is not specified, then the module errors out.

if ischar(inputVideo)
    % A path was passed in.
    % Read the video and once finished with this module, write the result.
    writeResult = true;
else
    % A video matrix was passed in.
    % Do not write the result; return it instead.
    writeResult = false;
end

Validate parameters and set defaults

Next, we need to get params fields for this module, their default values and validation functions. We can ValidateField.m function to check if values input to the function violate the constraints in any of the fields. If there is a missing field, this function adds that with the corresponding default value.

if nargin < 2 
    params = struct;
end

% validate params
[~,callerStr] = fileparts(mfilename);
[default, validate] = GetDefaults(callerStr);
params = ValidateField(params,default,validate,callerStr);

Of course, this assumes that you defined fields, default values, and validation functions for your new module inside the GetDefaults.m function. GetDefaults.m is one of the two utility functions in ReVAS (the other one is Filename.m but we will come to that later) that you need to modify to add/modify a given module. For the DummyModule.m example, we added the following case to GetDefaults.m.

function [default, validate, before, after, keyword, axesHandles] = GetDefaults(module)
. 
.
.
switch module
    
    case 'dummymodule'
        
        % default values
        default.overwrite = false;
        default.enableVerbosity = 'frame';
        default.badFrames = false;
        default.axesHandles = [];
        
        % validation functions 
        validate.overwrite = @islogical;
        validate.enableVerbosity = @(x) CategoryOrLogicalOrNumeric(x,{'none','video','frame'});
        validate.badFrames = @(x) all(islogical(x));
        validate.axesHandles = @(x) isempty(x) | all(ishandle(x));
        
        % list which modules can preceed or succeed this one
        before = {'degree2pixel','stripanalysis'};
        after = {'none','stripanalysis'};
        
        % keyword to be used in filenames
        keyword = 'dummy';
        
        % axes handle tags.. useful only in GUI mode
        axesHandles = {'imAx'};

  1. Let's break it down. The case name is simply the name of our module, all lowercase letters and without file extension, dummymodule.

  2. We have four required fields. We create two structures default and validate to store default values and validation functions.

  • overwrite: a logical flag indicating that the output file will be overwritten if it already exists. Defaults to false, i.e., do not overwrite if a file already exists.
   default.overwrite = false;
   validate.overwrite = @islogical;
  • enableVerbosity: a character array indicating the level of visualization requested. Defaults to frame meaning that we will get a visualization at every frame. Available modes are none (no visualization, and video (one visualization per video). Some other modules use simpler enableVerbosity fields which is simply a true/false flag.
   default.enableVerbosity = 'frame';
   validate.enableVerbosity = @(x) CategoryOrLogicalOrNumeric(x,{'none','video','frame'});
  • badFrames: A logical array indicating which frames are to be skipped during processing. Setting it to false is equivalent to false(numberOfFrames,1).
   default.badFrames = false;
   validate.badFrames = @(x) all(islogical(x));
  • axesHandles: An array to hold axes object(s) to be used for visualization. Defaults to [], empty. This is to make sure ReVAS does not crash when used independent from the GUI.
   default.axesHandles = [];
   validate.axesHandles = @(x) isempty(x) | all(ishandle(x));

ReVAS GUI uses additional output arguments of GetDefaults.m to properly populate axes handles for each module. To ensure that we create a variable axesHandles and set it to the name of one of the axes objects in ReVAS GUI.

   % axes handle tags.. useful only in GUI mode
   axesHandles = {'imAx'};

Available axes objects are given in the table below, numbers refer to the annotated image of the blank ReVAS GUI.

Number Axes Object
4 imAx
5 posAx
6 peakAx
7 motAx
8 stimAx
9 blinkAx

  1. We define the connectivity of this module with other modules.
   % list which modules can preceed or succeed this one
   before = {'none','trimvideo','stripanalysis'};
   after = {'none','stripanalysis'};
  • before is a cell array of module names. These modules can come before our DummyModule.m in the pipeline.
  • after is also a cell array of modules that can succeed our new module.

Adding 'none' to before means that this module can be the first module of a pipeline (i.e., there can be nothing before it). Likewise, adding 'none' to after means that this module can be the last module of a pipeline.

  1. Finally, we choose a keyword for our new module. This keyword will be appended to input file name when saving the output of our module. So choose something short but meaningful.

Handle GUI specific settings

When creating a module for ReVAS, keep in mind that it will be used both with and without GUI. In GUI mode, we use specific user interface elements; axes objects for visualization, text box objects for displaying messages/logs, push/toggle button objects to interrupt processing. A perfect UI design completely separates the GUI from actual processing routines. However, ReVAS is imperfect in the sense that depending on in which mode it operates (as a toolbox or an app with a GUI), it may need additional input. As of 5/11/2020, the only extra input ReVAS modules have to look for are logBox and abort fields under params structure. If a module is called from ReVAS GUI, the params structure will have these two fields to enable the module to write to log box (number 10 in the annotated figure above) and to allow user to prematurely end (abort) the processing by pressing a push button. This is handled with the code block below.

%% Handle GUI mode
% params can have a field called 'logBox' to show messages/warnings 
if isfield(params,'logBox')
    logBox = params.logBox;
    isGUI = true;
else
    logBox = [];
    isGUI = false;
end

% params will have access to a uicontrol object in GUI mode. so if it does
% not already have that, create the field and set it to false so that this
% module can be used without the GUI
if ~isfield(params,'abort')
    params.abort.Value = false;
end

Handle visualization options

Next, we handle visualization options and axes objects. Take a look at the following block of code.

%% Handle verbosity 
if ischar(params.enableVerbosity)
    params.enableVerbosity = find(contains({'none','video','frame'},params.enableVerbosity))-1;
end

% check if axes handles are provided, if not, create axes.
if params.enableVerbosity && isempty(params.axesHandles)
    fh = figure(2020);
    set(fh,'name','Dummy',...
           'units','normalized',...
           'outerposition',[0.16 0.053 0.4 0.51],...
           'menubar','none',...
           'toolbar','none',...
           'numbertitle','off');
    params.axesHandles(1) = subplot(1,1,1);
end

% clear axes
if params.enableVerbosity
    cla(params.axesHandles(1))
    tb = get(params.axesHandles(1),'toolbar');
    tb.Visible = 'on';
end

First part is an optional conversion of enableVerbosity parameter to a number. Second, we check if axesHandles is empty, and if so (which is usually the case in non-GUI usage of ReVAS), we create a figure and axes object. Finally, we clear the axes since same axes object might be used previously by another module and its contents may need to be cleared.

Overwrite scenarios

Last step before we get to the actual processing this module will perform, we need to handle overwrite scenarios. Take a look at the code below.

%% Handle overwrite scenarios.
if writeResult
    outputVideoPath = Filename(inputVideo, 'dummy');
    matFilePath = Filename(inputVideo, 'randomframe');
    params.outputVideoPath = outputVideoPath;
    params.matFilePath = outputVideoPath;
    
    if ~exist(outputVideoPath, 'file')
        % left blank to continue without issuing RevasMessage in this case
    elseif ~params.overwrite
        RevasMessage(['DummyModule() did not execute because it would overwrite existing file. (' outputVideoPath ')'], logBox);
        RevasMessage('DummyModule() is returning results from existing file.',logBox); 
        
        % try loading existing file contents
        load(matFilePath,'randomFrame','randomFrameNumber');
        params.randomFrame = randomFrame;
        if nargout > 2
            varargout{1} = randomFrameNumber;
        end
        return;
    else
        RevasMessage(['DummyModule() is proceeding and overwriting an existing file. (' outputVideoPath ')'], logBox);  
    end
else
    outputVideoPath = [];
    matFilePath = [];
end

Now, let's dissect this to understand what it does. First, if writeResult flag is set to false above in Determine input type, output of this module will not be written to a file and therefore, outputVideoPath is empty. On the other hand, if writeResult is true, then we append the input file path with our module's keyword (defined in GetDefaults.m) and extension by calling the Filename.m function. We also store this new path in a filed under params structure. We do the same for additional file (matFilePath) to be used for saving additional outputs. So far we covered the following lines.

if writeResult
    outputVideoPath = Filename(inputVideo, 'dummy');
    matFilePath = Filename(inputVideo, 'randomframe');
    params.outputVideoPath = outputVideoPath;
    params.matFilePath = outputVideoPath;
.
.
. 
    end
else
    outputVideoPath = [];
    matFilePath = [];
end

Now, let's diverge a little and see what we need add to Filename.m function for our module. Filename.m is the second of the two functions (the other one is GetDefaults.m) that must be modified for any new module. It can be used to (1) create new file paths for a given module, (2) get the keyword (suffix) of a module given its name, or (3) get the module name given its keyword. We added the following block of code to Filename.m for our DummyModule.m.

function [outputFilePath, keyword, module] = Filename(inputFilePath, moduleToApply, varargin)
.
.
.
switch lower(moduleToApply)
    case {'dummy','dummymodule'}
        keyword = 'dummy';
        module = 'dummymodule';
        outputFilePath = fullfile(inputDir, [inputFileName '_' keyword inputExtension]);
    case 'randomframe'
        keyword = 'randomframe';
        module = 'dummymodule';
        outputFilePath = fullfile(inputDir, [inputFileName '_' keyword '.mat']);

Note that only Hybrid modules need two separate case statements in Filename.m. One for primary output (e.g., video path) and another for saving additional outputs. Core and Bypass modules need only one case statement. In our example, we want DummyModule.m to return a randomly selected frame and its frame number in a separate file (when write out is enabled). We specify the file keyword (or suffix) for this additional output file in Filename.m function (not in GetDefaults.m). In a future release of ReVAS, these two functions may be consolidated into a single function. Check out RemoveStimuli.m and its entries in Filename.m and GetDefaults.m for another example.

Going back to handling overwrite scenarios, the next stop is check if a file exists with the full path specified in outputVideoPath. If it doesn't exist, proceed without doing anything else. If the file exists and overwrite field is set to true, print a message (to command line and also to logBox in GUI mode) since we will overwrite that file.

    if ~exist(outputVideoPath, 'file')
        % left blank to continue without issuing RevasMessage in this case
    elseif ~params.overwrite
    .
    .
    .
    else
        RevasMessage(['DummyModule() is proceeding and overwriting an existing file. (' outputVideoPath ')'], logBox);  
    end

Finally, the part where overwrite is set to false. This part should be handled with care, especially for Bypass and Hybrid modules, where in addition to the primary output, one or more additional outputs (also referred to as intermediate or auxiliary parameters) are returned. If outputVideoPath exists and overwrite is false, our module will simply load the contents of that file and return them instead. Before doing that, we also print out some messages to let the user know that results are loaded from an existing file.

        RevasMessage(['DummyModule() did not execute because it would overwrite existing file. (' outputVideoPath ')'], logBox);
        RevasMessage('DummyModule() is returning results from existing file.',logBox); 
        
        % try loading existing file contents
        load(matFilePath,'randomFrame','randomFrameNumber');
        params.randomFrame = randomFrame;
        if nargout > 2
            varargout{1} = randomFrameNumber;
        end
        
        return;

Note that if this was a Core module (such as TrimVideo.m or BandpassFilter.m), and if outputVideoPath exists and overwrite is false, we don't load the existing file and return that as an array. We simply return the outputVideoPath.

Main logic of the module

Here, we implement the operations that our module will perform. This part is completely up to the programmer/designer of the module, assuming that he/she will follow some basic programming guidelines (some specific to MATLAB). These guidelines are in short:

  • Use sensible variable names with Camel case.
  • Insert comments where necessary to explain what each part of the code does.
  • Break down code into smaller sections (using %% ) or sub-functions to improve readability and versatility.
  • Use vector operations as much as possible instead of for/while loops.
  • Use white space generously but cleverly to avoid clutter

The following code block shows the main logic of our DummyModule.m.

  • It creates video reader and writer objects (if necessary),
  • handles badFrames using a utility function of ReVAS HandleBadFrames.m (which is the method to be used to handle this field, if it exists, in any module),
  • selects a random number from 1:numberOfFrames,
  • goes over each frame, inverts pixel values, visualizes output if requested, and saves them in a new file or array,
  • assigns the randomly selected frame to a new array,
  • writes out randomFrame and randomFrameNumber in matFilePath and exits.
%% Create a reader object if needed and get some info on video

if writeResult
    writer = VideoWriter(outputVideoPath, 'Grayscale AVI');
    open(writer);

    % Determine dimensions of video.
    reader = VideoReader(inputVideo);
    params.frameRate = reader.FrameRate;
    numberOfFrames = reader.FrameRate * reader.Duration;
    
else
    [height, width, numberOfFrames] = size(inputVideo); 
    
    % preallocate the output video array
    outputVideo = zeros(height, width, numberOfFrames-sum(params.badFrames),'uint8');
end


%% badFrames handling
params = HandleBadFrames(numberOfFrames, params, callerStr);

%% select a frame randomly
randomFrameNumber = randi(numberOfFrames,1);

%% Write out new video or return a 3D array

% Read, invert, and write frame by frame.
for fr = 1:numberOfFrames
    if ~params.abort.Value

        % get next frame
        if writeResult
            frame = readFrame(reader);
            if ndims(frame) == 3
                frame = rgb2gray(frame);
            end
        else
            frame = inputVideo(:,:, fr);
        end

        % if it's a blink frame, skip it.
        if params.skipFrame(fr)
            continue;
        end

        % invert pixel values
        frame = 255 - frame;
        
        % get random frame if it is the right frame number
        if randomFrameNumber == fr
            randomFrame = frame;
        end

        % visualize
        if (params.enableVerbosity == 1 && fr == 1) || params.enableVerbosity > 1
            axes(params.axesHandles(1)); %#ok<LAXES>
            if fr == 1
                imh = imshow(frame,'border','tight');
            else
                imh.CData = frame;
            end
            title(params.axesHandles(1),sprintf('Inverting frames. %d out of %d',fr, numberOfFrames));
        end

        % write out
        if writeResult
            writeVideo(writer, frame);
        else
            nextFrameNumber = sum(~params.badFrames(1:fr));
            outputVideo(:, :, nextFrameNumber) = frame; 
        end
    else 
        break;
    end
    
    if isGUI
        pause(.02);
    end
end % end of video

Once completed, place your new module under core pipeline/ folder so that ReVAS recognizes it.

Writing a unit tester

Now that we have a new module, we need to write a unit tester for it to ensure its functionality. All unit testers are located under testing/ folder, their name must follow a specific convention: Testing_<module-name>.m, should not take any input arguments, and must return true or false. Below you can find the tester for our DummyModule.m.

function success = Tester_DummyModule

% suppress warnings
origState = warning;
warning('off','all');
success = true;

try
    %% read in sample video

    % the video resides under /testing folder.
    inputVideo = FindFile('aoslo.avi');

    %% First test
    % use default params
    p = struct; 
    p.overwrite = true;
    p.enableVerbosity = true;
    
    % test with a video path
    [outputVideoPath,p] = DummyModule(inputVideo, p); %#ok<*ASGLU>
    delete(p.matFilePath);
    
    %% Second test
    % test with a video array
    videoArray = ReadVideoToArray(inputVideo);
    p.badFrames = false(1,size(videoArray,3));
    p.badFrames([1 3 5]) = true;
    outputVideo = DummyModule(videoArray,p);
    
    % check if the difference in number of frames between two videos is
    % exactly equal to length of badFrames
    assert(size(videoArray,3) - size(outputVideo,3) == sum(p.badFrames));

    %% Third test
    % check if pixel value inversion works
    
    % read in inverted video, delete the file
    invertedVideoArray = ReadVideoToArray(p.outputVideoPath);
    delete(p.outputVideoPath);
    
    % check if sum of original video and inverted video is all 255
    assert(sum(sum((videoArray(:,:,2) + invertedVideoArray(:,:,2)) - uint8(255))) == 0);
    
catch
    success = false;
end

warning(origState);

To summarize what's going on in a typical unit tester:

  1. Suppress warnings and set success to true at the beginning.
  2. Run tests.
  • Use try/catch for running all tests, if any of them fails, set success to false in catch.
  • Separate different tests using %% and specify which scenario each test is for.
  • Use at least one demo video provided in demo/ folder to perform your tests.
  1. Restore warnings.

Note that unit testers are for ensuring functionality of a module when ReVAS is used as a toolbox. Writing testers for GUI based apps are far more advanced and not warranted for this project. For testing your module in GUI mode, the best method currently available is simply using ReVAS GUI, creating a new pipeline using New Pipeline Tool, adding your new module to a pipeline, and running it with a demo video. ReVAS GUI automatically handles generating sub-GUIs for adjusting parameters of all available modules. However, in some cases, modification of some of the GUI-related functions might be required. If this is the case for your module, please contact one of the maintainers of ReVAS repo (start with Mehmet).

ReVAS

Clone this wiki locally