Tp2000 819 importer refactor

[dougmills-DIT]

TP2000-667 - Importer refactor / rewrite

Why

Ultimately this work is required to fully hand over the Comm code automation piece to tariff managers without DE massaging every 5th import, plus the current importer is buggy, inflexible and has a cache that leaks.

• The caching implementation has been problematic, polluting future imports and failing repeat imports.
• Codebase is hard to manage, extend and debug, or even read.
• Child objects could be handled better
• Test coverage is poor, and hard to improve.
• No facility to report on import failures in the UI.
• Some objects are silently discarded (GoodsNomenclatureSuccessor)

What

The importer v2 addresses issues with the importer v1. Initially importer v2 will be added as an additional option within the footer menu, under "New TARIC parser" allowing the existing process to proceed as it did before, but allow the importer v2 to be employed. This will be for a time until confidence grows and its agreed that the importer v2 should replace importer v2 for comm code import capability.

• Removed the use of caching
  ** The implementation of caching was the cause if multiple issues.
  ** Cached data was not purged, and future imports could include previously cached data in certain circumstances.
  ** The need for caching was questionable to begin with, since a TARIC file can and should be processed in a linear way, each successive change should be built on the previous changes. The need for caching may have been due to the merging of certain data types (description and period for example) and the caching was needed when a change was presented to a child object that does not have a table to attach to until the parent object is created. This problem is addressed with the importer v2 by pre-processing child / parent objects and appending properties to the parent before commiting to the database. The only exception is when a child is explicitly deleted, without the subsequent deletion of the parent. This will raise a warning from the importer, but the whole import can proceed regardless, and the net effect is that the child record is not removed, unless the parent is removed. This is an unfortunate side effect of the choice to join these objects together in the database and unless we split these out to separate database tables, this possible edge case will exist.
• Coded in a more linear way, leading to better developer understanding and easier improvements to existing behaviour.
  ** The ability to extend behaviour is far easier and more accessible to understanding.
  ** Imports are not recursively handled like the importer v1, where a transaction object calls to a message object, that calls to a transmission object that calls to a record object, that is then identified as the data type - this process of parsing made debugging and altering the process very difficult and hard to see object properties and types.
  ** Its very easy to debug and see what's happening at any point.
  ** Stack traces make a lot more sense.
  ** Extending and changing the behaviour of an imported taric object, or a set of taric objects is far easier and simpler.
  ** The process is split into a validation stage, then a commit stage, depending on the validation passing.
• Better child objects handling (where a TARIC record is appended to a parent table), Where before changes would be lost or could not be processed correctly, now a more robust process around handling this type of update.
  ** There were circumstances where the importer v1 would simply discard a change, the importer v2 will not discard anything and either import or add a report item to the failed import with details about the failure.
  ** child parent relationships are validated and reported on if anything data can't be processed for a number of reasons.
  ** No data gets left behind!
• Very high test coverage for all TARIC types, and update types
  ** Unit Testing was a huge undertaking, to test every object type and every update type was a large task and very time consuming. It did however provide roughly 97% coverage on the new code.
  ** Integration testing at the time of development allowed us to evidence multiple examples of previous EU imports that failed, and required Data Engineering to massage the data when imported with importer v1, worked seamlessly with importer v2. This will have a huge impact on the comm code automation piece and truly allow us to hand over EU imports to the TIPS team.
• Reporting on import issues in the UI
  ** When an invalid scenario occurs, and data cant import correctly for one reason or another, there will be an error created against the import, available within the UI for a user to read and take action on.
  ** If an issue is detected that can be negated, the importer v2 will create a warning but will allow the import to complete. An example of this would be to delete a description period, since this is not a first class object in the TAP database, it can't be deleted without deleting the associated description. A warning will be recorded against the import, but will still allow the import to complete successfully, but provide the warning details in the importer UI.
• Importing all objects
  ** The importer v1 can miss objects and report success, we have seen this with both goods nomenclature origins and successors. the new importer imports EVERYTHING!

Checklist

• Requires migrations? YES
• Requires dependency updates? YES

Links to relevant material
See: JIRA Ticket