Skip to content

MFCS Roadmap:

Jump to bottom
David J. Davis edited this page May 5, 2021 · 10 revisions

ROADMAP FOR COMPLETING MFCS

How to use this document.

This document will be an explanation of all components and features recognized in the current version of MFCS. The contents will explain how it works currently in the ruby version and describe what needs to be done to finish the feature in the ruby version of MFCS. You can use the Table of Contents to get to linked content structures of the document.

TABLE OF CONTENTS

Administration Feature Set

  • System Tools
    • Toggles in and out of readonly mode which will allow you to read and search, but not modify any data.
  • User Administration
    • Creates a way to manage users allowed to access the system
    • System Users and OAuth (WVU Login)
  • Watermarking
    • Create, Read, Update, and Delete Watermarks for converted images.
  • Project Assignment
    • Create, Read, Update, and Delete of Project Tags for new Forms.
    • Allows project tags to be added to Items and Forms
    • Searchable Project tags for export

DATABASE STRUCTURES

  • The database structures are mostly the same, except for CamelCase values are being changed to snake_case.
  • The biggest change in structure is the hashing is now JSONB.
  • Performance Upgrades
    • FormBuilder Views (Use background jobs to make views of the form builder objects).
    • Create an abstract class that uses background view objects for search, view, etc.

Import

  • The database structures will need imported from MFCS (PROD), to MFCS (RUBY). To do this you will need to review the schema and make adjustments for the data as needed.
  • Once all schemas and models are configured to the production data, it would be best to use the models in an import script so that any and all callbacks are completed. This will ensure background jobs and enqueued and other processes are started (ex: conversion, archive, working files, etc).

Export

  • The database structures will need imported from MFCS (PROD), to MFCS (RUBY). To do this you will need to review the schema and make adjustments for the data as needed.
  • Once all schemas and models are configured to the production data, it would be best to use the models in an import script so that any and all callbacks are completed. This will ensure background jobs and enqueued and other processes are started (ex: conversion, archive, working files, etc).

Authentication (Database and CAS OAuth)

FROM / FIELD BUILDER

  • Arguably the most important part of MFCS.

Feature List

  • Create and manage Digital Object and Metadata Forms
  • Metadata Forms are essentially controlled vocabulary for various headings (subject, topical, etc)
  • Custom forms allow you to use any metadata schema for digital assets, or mix and match schemas
    • Allows you to add metadata mappings for exporting to match multiple schemas
  • Accessible Interface for both Digital Object Metadata form creation
  • Custom Fields
    • Apply CSS Classes, IDs, Local styles to form elements.
    • Set Default values with variable data (dates, times, etc)
    • Standard Validation
    • Complex Validation using Regular Expressions
    • Specify Required, Readonly, and other state specific displays an validations.
  • Unique IDNO numbers
    • System Created using a pattern ex test_###### would yield test_000001 extending the pattern as needed.
    • User supplied IDNO which checks against duplication

Status

  • Form builder is completed, however there may be unexpected bugs in building the form when edited, or when page is refreshed.
  • HTML/JS located in the app/frontend/components/form_builder
  • Stimulus JS located in app/frontend/controllers

Form Builder Navigation

  • This component of form builder is not complete. The interface currently establishes this as a list of items that is built along with the form using a tab system. I think this is poorly done because if the form is not accepted all the work will have to be redone. We need a better system for building form navigation. Possible need for drag and drop system created in the form builder too.

Templating

  • The form builder uses templates to keep down redundancy and to break up the HTML.

Preview Templates

  • The Form Builder is built using Stimulus JS, HTML, and Ruby.
  • Most of the HTML templates are used in the preview of the form builder in this aspect.

Field Templates

  • Location app/frontend/components/form_builder/fields
  • These are the interfaces for the administration who is creating the form, so these templates are what the user interacts with to determine which fields they are putting into the form. It also has the data-controllers for the stimulus that drives multiple interactions and building of data structures for the form submissions.
  • Complete, but not tested using BDD methods.
  • Needs tested using BDD because of the nature of these methods. There is a potential that switching these to view components we could avoid having to test with BDD methods, but it is doubtful because of JavaScript interactions that need to take place.

Field Templates

  • Location app/frontend/components/form_builder/fields
  • These are the interfaces for the administration who is creating the form, so these templates are what the user interacts with to determine which fields they are putting into the form. It also has the data-controllers for the stimulus that drives multiple interactions and building of data structures for the form submissions.
  • Complete, but not tested using BDD methods.
  • Needs tested using BDD because of the nature of these methods. There is a potential that switching these to view components we could avoid having to test with BDD methods, but it is doubtful because of JavaScript interactions that need to take place.

Field Builder Class

  • This is a series of modules and libraries to make a cleaner representation of the forms that are built for actually using the forms. So this instance will work with the logic provided form the JSON Data structures, parse it into manageable hashes, and use helpful classes to generate the HTML.
  • Complete, tested by Unit tests.
  • located in app/lib/field_builder

Digital Objects, Metadata Objects, Etc

  • Somewhat complete potential refactors and changes should happen.
  • /items/digital_objects shows the forms used for digital objects.
    • Shows Form
    • New (Makes a new item or digital object)
    • List Objects to allow for edit
    • Allow for deleting objects
  • /items/metadata shows the form used for metadata objects.
    • Shows Form
    • New (Makes a new item or digital object)
    • List Objects to allow for edit
    • Allow for deleting objects

Validation

IDNO Module

  • This is completed for system IDNO management, may need more work in the future.
  • IDNO is created using the record count, a padding size, and pattern replacement using substrings.
  • Race Conditions could possibly occur, if there is a bug this is the bug I would expect.

Processing Uploads

  • The Upload is able to process, but there is a bigger module in play with this. So while this small part of the process is done there are more that need to be added and modified before going into production. I would treat this as MVP and work on it after the beta release. Meaning that what is here is fine, but we should imitate chunking and abstract this to a AJAX upload so it doesn't block the form.

Permissions

  • This feature set is complete, but not added to the menu. It is able to be added and modified in it's own menu as well as under the forms in a much nicer way than it currently exists in MFCS. As an aside for this, the menu aspect needs to be added based on permissions.
  • Metadata / Forms / Objects.
    • These should only Show if the user has permissions to view, edit, modify, etc.
    • Simplify the interface, then it makes the application easier to use.

Item Search

  • This needs done, we need to create a search module system that uses the JSONB functions and potentially ElasticSearch to create an index to pull from, however PGSearch may get us by.

Watermarks

  • The watermarks is complete aside from the additions to the menu.

Menu Interface

  • The menu interface is incomplete, this is because the menu should be modified based on permissions.
  • I would handle the generation of the menu as a Module/Presenter that generates the HTML and Links based on the users permission levels.
  • Making a menu module would do a few things,
    1. this would make a way to test that the permission levels are what they say they are
    2. limits the number of links users have to look at which simplifies the application
    3. Make a hard coded object more maintainable.

Background Jobs

  • Background jobs are completed through the use of Redis and Sidekiq. While this adds to the total stack and size of the docker containers it adds a huge benefit to production by allowing us to run conversions as backgrounded tasks.

Media

  • Every copy of a media object is an associated database with a form, object, and has appropriate file information with it. This is so that the files are not just stored in the JSON of the item and have a way to be found, and eventually linked with any number of items.

File Conversions

  • The goal for this is to run a job that sets all the jobs in place for conversion of files. This way you can narrow down any failure points or errors using the sidekiq jobs interface to see failed jobs. As a side note you can copy that into an interface for all admins to see using an AJAX request or the documentation from sidekiq.

  • EX: If a user uploads 3 tiffs and an audio file. A job parser should run and create a background job for

    • Working Files / Archive Files
    • Conversions
    • OCR

  • This would setup a job queue that would look like

    • Working File Tif 1
    • Working File Tif 2
    • Working File Tif 3
    • Working File Audio File
    • Convert Tif 1
    • Convert Tif 2
    • Convert Tif 3
    • OCR TIF 1
    • OCR TIF 2
    • OCR TIF 3
    • Convert Audio
    • Combine PDF
    • Notify User (Conversion Complete) This can be done for each conversion job, or as a process set to send after the sequence of jobs are finished.

What does this solve?

  • This solves in the current version having to wait until 1:00am to see if a conversion is completed. It provides a better way to manage system resources without causing intentional delay. There could be problems associated with it such as incomplete jobs, a subsystem going down, race conditions, or un-synced source code however those risks are minimal and it would provide a better experience for the user.

Archive File

  • This is the file the user uploaded. Once uploaded it will be used once to create a working copy and will be stored as a preservation file only to be used if a working file doesn't exist.

Working Copy

  • A direct copy of the archive file that is moved into a working copy directory to ensure the archive file is not touched again. The base file for all uploaded versions.

Conversion Classes

  • The rest of these will be conversion types that are all classes to ensure that the modules can be re-used and are easily editable and will deal with possible conditions of the conversions and different options.
  • Image
  • Audio
  • Video
  • OCR

Ocr

  • OCR Library has 2 options, saving a PDF Copy of the OCR Document
  • Returning Text to save in the database along with that media object.
  • Currently, I think we should do both. Save a copy of the text in the media Object and

Project Tagging

  • This basically assigns a tag or search queue for finding a resource. These can be applied to users, items, and forms, but I think currently it only functions with forms in the form of the tag being applied to an item. There has to be a better way to get this relationship established and then query on it.

Notifications

  • Currently the notifications system in MFCS tells you when an automatic export is complete. I think I would expand on this to do in browser notifications for things when a User is logged in and also using the mailer functionality.

Documented and Built By WVU Libraries. If the content in this page is outdated please let us know.

Please Submit Github Issues for the following types of requests:

  • Documentation Update
  • Bug Fixes
  • Feature Enhancement
  • New Features
Clone this wiki locally