Remove django-import-export

docs/docs/settings/export.md

@@ -4,14 +4,10 @@ title: Exporting Data
 
 ## Exporting Data
 
-The Admin Interface provides powerful data exporting capability. When displaying a list of items which support exporting (e.g. Part objects), select the "Export" button from the top-right corner:
+InvenTree provides data export functionality for a variety of data types. Most data tables provide an "Export Data" button, which allows the user to export the data in a variety of formats.
 
-{% with id="export", url="admin/export.png", description="Data export" %}
-{% include 'img.html' %}
-{% endwith %}
-
-Multiple data formats are supported for exported data:
+In the top right corner of the table, click the "Export Data" button to export the data in the table.
 
-{% with id="formats", url="admin/formats.png", description="Data formats" %}
+{% with id="export", url="admin/export.png", description="Export data" %}
 {% include 'img.html' %}
 {% endwith %}

docs/docs/settings/import.md

@@ -15,77 +15,70 @@ External data can be imported via the admin interface, allowing for rapid integr
 !!! warning "Supported Models"
     Not all models in the InvenTree database support bulk import actions.
 
-When viewing a model (which supports bulk data import) in the admin interface, select the "Import" button in the top-right corner:
+### Required Permissions
 
-{% with id="import", url="admin/import.png", description="Data import" %}
-{% include 'img.html' %}
-{% endwith %}
+To import data, the user must have the appropriate permissions. The user must be a *staff* user, and have the `change` permission for the model in question.
 
-The next screen displays a list of column headings which are expected to be present in the uploaded data file.
+## Import Session
 
-{% with id="import_upload", url="admin/import_upload.png", description="Data upload" %}
-{% include 'img.html' %}
-{% endwith %}
+Importing data is a multi-step process, which is managed via an *import session*. An import session is created when the user initiates a data import, and is used to track the progress of the data import process.
 
-Select the data file to import, and the data format. Press the "Submit" button to upload the file.
+### Import Session List
 
-### File Format
+The import session is managed by the InvenTree server, and all import session data is stored on the server. As the import process can be time-consuming, the user can navigate away from the import page and return later to check on the progress of the import.
 
-The uploaded data file must meet a number of formatting requirements for successful data upload. A simple way of ensuring that the file format is correct is to first [export data](./export.md) for the model in question, and delete all data rows (not the header row) from the exported data file.
+Import sessions can be managed from the *Admin Center* page, which lists all available import sessions
 
-Then, the same file can be used as a template for uploading more data to the server.
+### Context Sensitive Importing
 
-### ID Field
+Depending on the type of data being imported, an import session can be created from an appropriate page context in the user interface. In such cases, the import session will be automatically linked to the relevant data type being imported.
 
-The uploaded data file requires a special field called `id`. This `id` field uniquely identifies each entry in the database table(s) - it is also known as a *primary key*.
+## Import Process
 
-The `id` column **must** be present in an uploaded data file, as it is required to know how to process the incoming data.
+The following steps outline the process of importing data into InvenTree:
 
-Depending on the value of the `id` field in each row, InvenTree will attempt to either insert a new record into the database, or update an existing one.
+### Create Import Session
 
-#### Empty ID
+An import session can be created via the methods outlined above. The first step is to create an import session, and upload the data file to import. Note that depending on the context of the data import, the user may have to select the database model to import data into.
 
-If the `id` field in a given data row is empty (blank), then InvenTree interprets that particular row as a *new* entry which will be inserted into the database.
-
-If you wish for a new database entry to be created for a particular data row, the `id` field **must** be left blank for that row.
+{% with id="import-create", url="admin/import_session_create.png", description="Create import session" %}
+{% include 'img.html' %}
+{% endwith %}
 
-#### Non-Empty ID
+### Map Data Fields
 
-If the `id` field in a given data row is *not* empty, then InvenTree interprets that particular row as an *existing* row to override / update.
+Next, the user must map the data fields in the uploaded file to the fields in the database model. This is a critical step, as the data fields must be correctly matched to the database fields.
 
-In this case, InvenTree will search the database for an entry with the matching `id`. If a matching entry is found, then the entry is updated with the provided data.
+{% with id="import-map", url="admin/import_session_map.png", description="Map data fields" %}
+{% include 'img.html' %}
+{% endwith %}
 
-However, if an entry is *not* found with the matching `id`, InvenTree will return an error message, as it cannot find the matching database entry to update.
+The InvenTree server will attempt to automatically associate the data fields in the uploaded file with the database fields. However, the user may need to manually adjust the field mappings to ensure that the data is imported correctly.
 
-!!! warning "Check id Value"
-    Exercise caution when uploading data with the `id` field specified!
+### Import Data
 
-### Import Preview
+Once the data fields have been mapped, the data is loaded from the file, and stored (temporarily) in the import session. This step is performed automatically by the InvenTree server once the user has confirmed the field mappings.
 
-After the data file has been uploaded and validated, the user is presented with a *preview* screen, showing the records that will be inserted or updated in the database.
+Note that this process may take some time if the data file is large. The import process is handled by the background worker process, and the user can navigate away from the import page and return later to check on the progress of the import.
 
-Here the user has a final chance to review the data upload.
+### Process Data
 
-Press the *Confirm Import* button to actually perform the import process and commit the data into the database.
+Once the data has been loaded into the import session, the user can process the data. This step will attempt to validate the data, and check for any errors or issues that may prevent the data from being imported.
 
-{% with id="import_preview", url="admin/import_preview.png", description="Data upload preview" %}
+{% with id="import-process", url="admin/import_session_process.png", description="Process data" %}
 {% include 'img.html' %}
 {% endwith %}
 
-Note that *new* records are automatically assigned an `id` value.
-
-## Import Errors
+Note that each row must be selected and confirmed by the user before it is actually imported into the database. Any errors which are detected will be displayed to the user, and the user can choose to correct the data and re-process it.
 
-Manually importing data in a relational database is a complex process. You may be presented with an error message which describes why the data could not be imported.
+During the processing step, the status of each row is displayed at the left of the table. Each row can be in one of the following states:
 
-The error message should contain enough information to manually edit the data file to fix the problem.
+- **Error**: The row contains an error which must be corrected before it can be imported.
+- **Pending**: The row contains no errors, and is ready to be imported.
+- **Imported**: The row has been successfully imported into the database.
 
-Any error messages are displayed per row, and you can hover the mouse over the particular error message to view specific error details:
-
-{% with id="import_error", url="admin/import_error.png", description="Data upload error" %}
-{% include 'img.html' %}
-{% endwith %}
+Each individual row can be imported, or removed (deleted) by the user. Once all the rows have been processed, the import session is considered *complete*.
 
+### Import Completed
 
-!!! info "Report Issue"
-    If the error message does not provide enough information, or the error seems like a bug caused by InvenTree itself, report an [issue on Github](https://github.com/inventree/inventree/issues).
+Once all records have been processed, the import session is considered complete. The import session can be closed, and the imported records are now stored in the database.

src/backend/InvenTree/InvenTree/admin.py

@@ -1,132 +1,10 @@
 """Admin classes."""
 
 from django.contrib import admin
-from django.db.models.fields import CharField
 from django.http.request import HttpRequest
 
 from djmoney.contrib.exchange.admin import RateAdmin
 from djmoney.contrib.exchange.models import Rate
-from import_export.exceptions import ImportExportError
-from import_export.resources import ModelResource
-
-
-class InvenTreeResource(ModelResource):
-    """Custom subclass of the ModelResource class provided by django-import-export".
-
-    Ensures that exported data are escaped to prevent malicious formula injection.
-    Ref: https://owasp.org/www-community/attacks/CSV_Injection
-    """
-
-    MAX_IMPORT_ROWS = 1000
-    MAX_IMPORT_COLS = 100
-
-    # List of fields which should be converted to empty strings if they are null
-    CONVERT_NULL_FIELDS = []
-
-    def import_data_inner(
-        self,
-        dataset,
-        dry_run,
-        raise_errors,
-        using_transactions,
-        collect_failed_rows,
-        rollback_on_validation_errors=None,
-        **kwargs,
-    ):
-        """Override the default import_data_inner function to provide better error handling."""
-        if len(dataset) > self.MAX_IMPORT_ROWS:
-            raise ImportExportError(
-                f'Dataset contains too many rows (max {self.MAX_IMPORT_ROWS})'
-            )
-
-        if len(dataset.headers) > self.MAX_IMPORT_COLS:
-            raise ImportExportError(
-                f'Dataset contains too many columns (max {self.MAX_IMPORT_COLS})'
-            )
-
-        return super().import_data_inner(
-            dataset,
-            dry_run,
-            raise_errors,
-            using_transactions,
-            collect_failed_rows,
-            rollback_on_validation_errors=rollback_on_validation_errors,
-            **kwargs,
-        )
-
-    def export_resource(self, obj):
-        """Custom function to override default row export behavior.
-
-        Specifically, strip illegal leading characters to prevent formula injection
-        """
-        row = super().export_resource(obj)
-
-        illegal_start_vals = ['@', '=', '+', '-', '@', '\t', '\r', '\n']
-
-        for idx, val in enumerate(row):
-            if type(val) is str:
-                val = val.strip()
-
-                # If the value starts with certain 'suspicious' values, remove it!
-                while len(val) > 0 and val[0] in illegal_start_vals:
-                    # Remove the first character
-                    val = val[1:]
-
-                row[idx] = val
-
-        return row
-
-    def get_fields(self, **kwargs):
-        """Return fields, with some common exclusions."""
-        fields = super().get_fields(**kwargs)
-
-        fields_to_exclude = ['metadata', 'lft', 'rght', 'tree_id', 'level']
-
-        return [f for f in fields if f.column_name not in fields_to_exclude]
-
-    def before_import(self, dataset, using_transactions, dry_run, **kwargs):
-        """Run custom code before importing data.
-
-        - Determine the list of fields which need to be converted to empty strings
-        """
-        # Construct a map of field names
-        db_fields = {field.name: field for field in self.Meta.model._meta.fields}
-
-        for field_name, field in self.fields.items():
-            # Skip read-only fields (they cannot be imported)
-            if field.readonly:
-                continue
-
-            # Determine the name of the associated column in the dataset
-            column = getattr(field, 'column_name', field_name)
-
-            # Determine the attribute name of the associated database field
-            attribute = getattr(field, 'attribute', field_name)
-
-            # Check if the associated database field is a non-nullable string
-            if (
-                (db_field := db_fields.get(attribute))
-                and (
-                    isinstance(db_field, CharField)
-                    and db_field.blank
-                    and not db_field.null
-                )
-                and column not in self.CONVERT_NULL_FIELDS
-            ):
-                self.CONVERT_NULL_FIELDS.append(column)
-
-        return super().before_import(dataset, using_transactions, dry_run, **kwargs)
-
-    def before_import_row(self, row, row_number=None, **kwargs):
-        """Run custom code before importing each row.
-
-        - Convert any null fields to empty strings, for fields which do not support null values
-        """
-        for field in self.CONVERT_NULL_FIELDS:
-            if field in row and row[field] is None:
-                row[field] = ''
-
-        return super().before_import_row(row, row_number, **kwargs)
 
 
 class CustomRateAdmin(RateAdmin):

src/backend/InvenTree/InvenTree/settings.py

@@ -282,7 +282,6 @@
     'django_filters',  # Extended filter functionality
     'rest_framework',  # DRF (Django Rest Framework)
     'corsheaders',  # Cross-origin Resource Sharing for DRF
-    'import_export',  # Import / export tables to file
     'django_cleanup.apps.CleanupConfig',  # Automatically delete orphaned MEDIA files
     'mptt',  # Modified Preorder Tree Traversal
     'markdownify',  # Markdown template rendering
@@ -1052,9 +1051,6 @@
 
 DATE_INPUT_FORMATS = ['%Y-%m-%d']
 
-# Use database transactions when importing / exporting data
-IMPORT_EXPORT_USE_TRANSACTIONS = True
-
 # Site URL can be specified statically, or via a run-time setting
 SITE_URL = get_setting('INVENTREE_SITE_URL', 'site_url', None)