Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
TP2000-1140 - Asynchronous measure bulk creation (#1129)
* TP2000 1147 bulk edit celery task (#1124) * adding infra for new celery task * bulk_handling celery task * TP2000 1119 create measure command class (#1127) * Form mixin and adapt view. * adding command class CreateMeasures; migration for same * adding infra for new celery task * bulk_handling celery task * bulk_handling celery task * TP2000-1142 Synchronous only operation errors (#1114) * Updated NIG5 to correctly handle only goods create and update and NIG5_origin to only handle origin deletions (#1116) Also added NIG11 skip delete * TP2000-1160 Fix publish_to_api task envelope ID error (#1118) * Use seed envelope ID env variable * Remove int type conversion from seed envelope ID setting value * Update to remove validate_taric_xml_record_order as a requirement for export / publishing of TARIC data (#1119) * Quota origins and exclusions geo area description query optimisation (#1117) * Optimise geo area description queries * Add tests * Tp2000 819 importer refactor (#1015) * initial new parser commit * add new parsers and process for getting info from xml to dictionary for message * updated to using subclasses to find all parsers, then populate the object from the data provided * updated / added new parsers for all objects * updated parsers and linked to handlers (for dependency data, may refactor) * added validation + working on reporting issues * TP2000-844 Add suffix and indent columns to find commodities view (#885) * Add suffix and indent column * Update test * Show indent as at today * Update test * Refactor * Use optional type hint * TP2000-738 Display rule check progress (#887) * Display rule check progress in UI * Add test * Add test docstring * TP2000-849 Add active state filter to quota search (#886) * Add active state filter * Add test * Test invalid active state filter option * Rename fixture variable * TP2000-741 ME32 & ME16 rules checker failing (#874) * TP2000-477 Prepopulate geo exclusions (#889) * Pre-populate geo exclusions formset * Add conditions initial data, update tests * Fix tests * Fix tests * Add test for formset_add_or_delete * Add erga omnes prefix * Use regex * Tp2000 762 virus checker (#888) * adding docker network * virus check validator * skip virus check localy * removing variable * validator migration * PR comments and handler update * updating to make it obvious creds aren't real * removing dev file upoad handler * testing defaults * updated local config * TP2000-806 get envelope history (#890) * TP2000-860 Sqlite dump and upload failing (#892) * Skip migration during sqlite database creation. * Run Sqlite export process during out of hours. * updated exclusion mapping list as country has none (#895) * TP2000-861 Fix footnote delete button on measure edit view (#894) * Fix delete button * Add trailing slash to auto-generated URLs * Order indents to ensure access to latest version. (#897) * updating link data (WIP) * updating link data (WIP) * updating link data (WIP) * set parent import model (WIP) * set parent import model (WIP) * WIP - adding tests with example envelopes for the new importer * wip * wip * initial new parser commit * add new parsers and process for getting info from xml to dictionary for message * updated to using subclasses to find all parsers, then populate the object from the data provided * updated / added new parsers for all objects * updated parsers and linked to handlers (for dependency data, may refactor) * added validation + working on reporting issues * updating link data (WIP) * updating link data (WIP) * updating link data (WIP) * set parent import model (WIP) * set parent import model (WIP) * WIP - adding tests with example envelopes for the new importer * wip * wip * test updates * test updates * test updates * test updates * test updates * test updates * test updates * test updates * test updates * test updates * test updates * test updates * test updates * test updates * building out some UI for the new importer * building out some UI for the new importer * building out some UI for the new importer * building out some UI for the new importer * adding identity fields to all parsers * fix flaky tests * testing updates * testing updates * testing updates - finished update tests for all TARIC objects we use currently * testing updates - added add codes delete tests * testing updates - added add codes delete tests * testing updates - added certificate, commodity, footnote, geoarea, and measure delete tests * moved new parser / importer to its own app * finished adding delete tests and removed unused methods * added documentation * added documentation + small fixes * added documentation + small fixes * added documentation + small fixes * update comm code import objects * added documentation + small fixes * added documentation + small fixes * added documentation + small fixes * minor updates based on PR comments * minor updates based on PR comments * minor updates based on PR comments * minor updates based on PR comments * minor updates based on PR comments * minor updates based on PR comments * test fix for py 3.8 * test fix for py 3.8 * tidy up chunker a little and add tests * duplicated a bunch of files referenced in the importer app - anything thats not a model now also lives in the taric_parsers app. * duplicated a bunch of files referenced in the importer app - anything that's not a model now also lives in the taric_parsers app. * minor changes from PR review * updated issue_type for import error creation to a text choice * added tests to UI elements * altered ot absolute imports * altered exception raised on TaricXmlSourceBase * import tidy up - absolute imports * minor updates * minor updates * minor updates * minor updates * minor updates * minor updates * minor updates * minor updates * Updating documentation + PR feedback * Updating documentation + PR feedback * updates based on PR feedback * updates based on PR feedback * updates based on PR feedback * updates based on PR feedback * updates based on PR feedback * updates based on PR feedback * updates based on PR feedback * updates based on PR feedback * updates based on PR feedback * updates based on PR feedback * updates based on PR feedback * updates based on PR feedback * updates based on PR feedback * update pre-commit-hook format, was showing as invalid yaml - no functional change - just formatting * updates to readme, adding pre-commit info * updates to readme, adding pre-commit info * updates to readme, adding pre-commit info * updates to readme, adding pre-commit info * updates to readme, adding pre-commit info * rename to remove New prefix * rename to remove New prefix * PR updates * PR updates * PR updates * PR updates - adding in importerv2 to main comm code UI * PR updates : integrating UI - mid progress * minor UI update * fix unit tests mid way * interim commit * interim commit * interim commit * interim commit * fixed tests * fixed tests * fixed tests * add in missing migration * update NIG5 to not trigger when goods nomenclature also being deleted in the same transaction --------- Co-authored-by: Dale Cannon <[email protected]> Co-authored-by: Paul Pepper <[email protected]> Co-authored-by: Edie Pearce <[email protected]> Co-authored-by: Anthoni Gleeson <[email protected]> * feat: Improvement tweaks for creating quota definition periods (#1120) * feat: Allow table reports to be exported as a CSV (#1122) * feat: Quotas Definitions' That Are Expiring Soon (#1121) * Bump jinja2 from 3.1.2 to 3.1.3 (#1126) Bumps [jinja2](https://github.com/pallets/jinja) from 3.1.2 to 3.1.3. - [Release notes](https://github.com/pallets/jinja/releases) - [Changelog](https://github.com/pallets/jinja/blob/main/CHANGES.rst) - [Commits](pallets/jinja@3.1.2...3.1.3) --- updated-dependencies: - dependency-name: jinja2 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * empty * empty * empty --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Paul Pepper <[email protected]> Co-authored-by: Paul Pepper <[email protected]> Co-authored-by: Doug Mills <[email protected]> Co-authored-by: Dale Cannon <[email protected]> Co-authored-by: Edie Pearce <[email protected]> Co-authored-by: Anthoni Gleeson <[email protected]> Co-authored-by: Tash Boyse <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * TP2000-1172 Some measure form serialization / deserialization (#1139) * Refactor and SerializableFormMixin related work. * Move import to top. * Add docstring. * Always return an iterable (list) object. * DateRange serialization and type hinting. * Type hint fix and debug prints. * Corrected form attr and date range access. * Get serialized form data in view. * Deserialization and date handling. * Form reconstruction and validation. * Simplify form serialization. * Debug assistance and form serialization. * Simplify and clean up. * Cleanup. * Serialize quota order number and add code forms. * Better docstring. * Save kwargs and transaction on bulk creator and use it. * Extend SerializableFormMixin interface. * Add use of form kwargs when saving a MeasuresBulkCreator. * Add log info and docstring. * Add utility fn and docstring. * Remove unused type. * Improved strategy for filtering in serialized data. * More logging. * TP2000-1182 Serialize footnotes and conditions formsets (#1141) * MeasureFootnotesFormSet serialization. * Move mixin to top of file. Add support for conditions. * Add support for conditions. * TP2000-1179 Serialize commodities and duties formset (#1144) * Increased flexibility of form data key filtering and test. * Corrected logging. * Date serialization and tests. * Commodities and duties formset serialization. * Move import to top and add docstring. * Unit test MeasureCommodityAndDutiesFormSet serialization / deserialization. * Better form prefix handling and test update. * Fix incorrect use of serializable(). * Remove and replace test list with data form list. * Use new data_form_list. * Docstring tweak. * Docstring tweak. * Remove debug output. * More log output. * Concatenate lists Pythonically. * Inherit SerializableFormMixin on MeasureGeographicalAreaForm. * Naming refactor to `serializable_data()` and docstring improvements. * Fix typo. * TP2000-1178 Serialize geo areas form (#1147) * Support sync/async measure creation option. * Factor out queryset filtering and form.initial setup. * Apply form fixup for * Perculiar geo_areas prefix Form support. * Fix missing date deserialization. * TP2000-1148 Create asynchronous confirmation page (#1151) * TP2000-1222 Refactor measures based models (#1153) * Refactor measures models into separate files. * Corrected typo in docstring. * TP2000 1175 Measure Create Wizard Forms Serialization and Deserialization tests (#1157) * Refactor and SerializableFormMixin related work. * Move import to top. * Add docstring. * Always return an iterable (list) object. * DateRange serialization and type hinting. * Type hint fix and debug prints. * Corrected form attr and date range access. * Get serialized form data in view. * Deserialization and date handling. * Form reconstruction and validation. * Simplify form serialization. * Debug assistance and form serialization. * Simplify and clean up. * Cleanup. * Serialize quota order number and add code forms. * Better docstring. * Add initial form serializable tests * Remove type checking in test * Add additional codes form test * Add quota order number form tests * Get form to be valid in quota origins test - test not complete * Refactor tests and add selectable objects form test * Add deserialization of measure_start_date to conditions wizard form * Add parameterized test for formset serialization - Conditions * Refactor test and add footnotes fixture * WIP - Geo Areas test - does not work * Remove geoareas test and clean up * Remove unnecessary new line * Add doc string explaining separation --------- Co-authored-by: Paul Pepper <[email protected]> * TP2000-1207 Create measures from reconstructed forms (#1156) * Add model manager and processing infra code. * Use MeasuresBulkCreatorManager on model class objects. * Fixed typos and removed call to undefined function. * Factor out task scheduling into a schedule() member function. * Factor out creator behaviour from View. * Remove migration resulting from incorrect Model inheritance. * Factor out measures creation behaviour from wizard view class. * Improve log output. * New model attrs. Refactor measure creation behaviour. * Add trailing slash to URL. * Create MeasuresBulkCreators with a workbasket param. Added missing template name to view. * Export ProcessingState from models. * Unit test for measures wizard confirm view. * Reformat docstring. * Amend TODO comment. * Factory for MeasuresBulkCreator. * Fixture to mock apply_async call on bulk_create_measures. * Use synchronous approach to creating measures. * Reinstate create_measures() for unit test. * Unit test bulk processing. * Exclude form control data from serialization. * Fixed typo. * Tp2000 1199 test measure geo area serialization (#1162) * Add comments for Edie * Add working Geo areas test * Parameterize measures geo area serialization test * Add erga-omnes-exclusions param to geo areas test * Add geo group and exclusions fixtures to test * TP2000-1237 - processing_state as FSMField (#1168) * Move fixtures and add new tests. * Add tests. * New migration adding processing_state model field attr. * Apply renamed function. * Tweak docstring and remove unnecessary test. * Add docstring and logging. * Manage processing_state in task execution and add logging. * Add type hints. * Remove redundant model BulkProcessorResult. * Elaborate TODO. * Add user field attr to MeasuresBulkCreator. Catch form errors in expected_measures_count. * Migration for new user field attr. * Model manager change, adding user param. * Remove TODO comment. * Tp2000 1149 new measures edit list view (#1169) * Tp2000-1243 Cancel task journey (#1174) * Implement cancel task journey. * Fix merge breakage. * Fix after merge breakage. * Fix after merge breakage. * Set celery process and queue names. * Support for user displayname and tests. * Use User.get_displayname() to show a user's name. * TP2000-1239 - measures creation progress counter (#1179) * Add model attribute to save number of successfully processed objects (measures). * Display item count and add unit tests. * Set `successfully_processed_count` on measures creation. * TP2000-1119 List view filtering (#1176) * formatting * except PROCESSING, filtering and selected_link working; tests wip, TODO: filter PROCESSING and tidy tests * PROCESSING filter, tests * added assert not on tests; removing comments * tidyup * refactor of filter query * TP2000-1254 Set Celery concurrency to 1 (#1188) * Set Celery concurrency level to 1 * Temp commit for testing. * Test: temporarily permit repeat measures creation. * Add to-do * Remove test code. * Set concurrency to 1 for bulk-create-worker. * Set rule-check container's Celery concurrency to 1. * Reinstate spurious deletion. * TP2000-1248 Bulk Create Measure journey UI changes (#1189) * Change object count to Item count * Change filter text from cancelled to terminated * Change action link and associated functions from cancelled to terminate * Change status badge from cancelled to terminated * Add functionality to show action as terminated for cancelled tasks * Amend tests broken from changing cancelled to terminated * Change the title on the cancel page to terminate measure process * Amend table and details header to match similar patterns in the service * change expected object count to expected item count * Change copy on the bulk measure creation termination page * Change warning text from cancelled to terminated * Change cancel button to terminate measure process * rename page and fix breadcrub to termination request sent * Change return button copy * Make the button sit where we want * TP2000-1244 Squashing migrations (#1192) * deleting old migrations and creating new single migrations * missing import in measures/views.py * Get current transaction from workbasket. (#1194) * squashing migrations (#1197) --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Charlie Prichard <[email protected]> Co-authored-by: Lauren Qurashi <[email protected]> Co-authored-by: Lauren Qurashi <[email protected]>
- Loading branch information
4 people
authored
Apr 19, 2024
1 parent
a4f7f31
commit e4d25d2
