Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Metadata uploader batch endpoints #1408

Merged
merged 60 commits into from
Jan 10, 2023
Merged
Show file tree
Hide file tree
Changes from 52 commits
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
2e25abb
starting to create new batch api endpoints
ksierks Nov 4, 2022
c186502
Merge branch 'metadata-uploader-base' into metadata-uploader-batch-en…
ksierks Nov 4, 2022
9f9a7f7
sending requests in chunks
ksierks Nov 9, 2022
6aa9d6c
fixing error handling
ksierks Nov 9, 2022
bdec14e
Merge branch 'metadata-uploader-base' into metadata-uploader-batch-en…
ksierks Nov 10, 2022
1ee66b0
starting to add unique key on label for metadata_field table
ksierks Nov 15, 2022
23aaf61
Merge branch 'metadata-uploader-base' into metadata-uploader-batch-en…
ksierks Nov 23, 2022
6e72446
adding promise chaining back in
ksierks Nov 25, 2022
211152d
moving returning ResponseEntity into controller
ksierks Dec 5, 2022
05b1340
fixing broken test
ksierks Dec 5, 2022
d9bb07f
fixing create project sample from modal
ksierks Dec 5, 2022
8a53a63
moving returning ResponseEntity into controller
ksierks Dec 5, 2022
60e4bec
adding more tests
ksierks Dec 5, 2022
8845722
adding comments
ksierks Dec 5, 2022
bb75d78
Merge branch 'metadata-uploader-base' into metadata-uploader-batch-en…
ksierks Dec 5, 2022
d9f98a8
fixing unmounted component error
ksierks Dec 5, 2022
79d4c90
changing path names
ksierks Dec 7, 2022
b357be2
returning a mult-status response from controller
ksierks Dec 7, 2022
43780af
adding functionality to calculate chunk size
ksierks Dec 7, 2022
a059c96
creating CreateSampleResponse and UpdateSampleResponse objects
ksierks Dec 8, 2022
c162ee4
Merge branch 'metadata-uploader-base' into metadata-uploader-batch-en…
ksierks Dec 8, 2022
983aa63
fixing tests
ksierks Dec 8, 2022
3576845
playing with a second web driver to have seperate users logged in sim…
ksierks Dec 9, 2022
2993915
cleaning up unused, old code
ksierks Dec 9, 2022
2f65a56
setting up delete samples during import test
ksierks Dec 12, 2022
07a587b
setting up create new sample during import test
ksierks Dec 12, 2022
e2c6c7f
setting up remove access during import test
ksierks Dec 12, 2022
ac0fdaa
finishing tests
ksierks Dec 16, 2022
4a394f0
updating documentation
ksierks Dec 16, 2022
ad886b2
Merge branch 'metadata-uploader-base' into metadata-uploader-batch-en…
ksierks Dec 16, 2022
deca056
trying to fix flaky tests
ksierks Dec 19, 2022
84c31f2
trying to fix flaky tests
ksierks Dec 19, 2022
874f6ab
trying to fix flaky tests
ksierks Dec 20, 2022
eb581b8
removing old restriction code
ksierks Dec 21, 2022
d0b3495
adding more comments
ksierks Dec 21, 2022
4712e69
using metadataSaveDetails from state
ksierks Dec 21, 2022
3eda36c
removing log statements
ksierks Dec 21, 2022
8f6bde8
adding partial success upload notification
ksierks Dec 21, 2022
8665d0c
removing returned metadataSaveDetails as it is redundant
ksierks Dec 21, 2022
687f4bf
creating a import-utilities.js file for refactoring
ksierks Dec 21, 2022
6ee2590
simplifying code and calling Promise.all
ksierks Dec 21, 2022
0d84b2c
using forEach instead of map
ksierks Dec 21, 2022
54cc39e
returning more explicit filter
ksierks Dec 21, 2022
03e22a8
adding comments
ksierks Dec 22, 2022
9071f0c
more refactoring
ksierks Dec 22, 2022
2cd44f6
more refactoring
ksierks Dec 23, 2022
63ca764
fixing some tests
ksierks Jan 4, 2023
5d7b998
Merge branch 'metadata-uploader-base' into metadata-uploader-batch-en…
ksierks Jan 6, 2023
47a06aa
fixing chunkArray function
ksierks Jan 6, 2023
234c460
fixing chunkArray function
ksierks Jan 6, 2023
a62081d
taking care of all the 'any' types
ksierks Jan 6, 2023
b7cbb89
memorizing filter
ksierks Jan 6, 2023
600dcbd
applying eslint suggestions
ksierks Jan 6, 2023
2a6ad22
making a clone of param because splice is destructive
ksierks Jan 6, 2023
c391d57
resolving typescript warning
ksierks Jan 6, 2023
3ce8e16
resolving typescript warning
ksierks Jan 6, 2023
8a388a5
renaming type
ksierks Jan 6, 2023
7931412
fixing tests
ksierks Jan 7, 2023
e17ee8d
applying eslint suggestion
ksierks Jan 9, 2023
7fd0d33
renaming SampleItemResponse
ksierks Jan 9, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added doc/user/user/sample-metadata/images/loading.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file not shown.
Binary file modified doc/user/user/sample-metadata/images/upload-preview-errors.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified doc/user/user/sample-metadata/images/upload-preview-success.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified doc/user/user/sample-metadata/images/upload-preview.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified doc/user/user/sample-metadata/images/upload-selection.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
20 changes: 14 additions & 6 deletions doc/user/user/sample-metadata/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,23 +37,31 @@ Links to the upload page can be found:

Any CSV or Excel spreadsheet containing metadata for samples in a project can be uploaded through the IRIDA web interface. One of the column in the table __must__ correspond to the sample name within the project. In this example spreadsheet, the `NLEP #` column is the sample name.

The first step is to select the CSV or Excel file containing the data. Either click on the square label `Click or drop Excel/CSV file containing metadata for samples in this project.` or drag and drop the file from your file browser.
The first step is to select the CSV or Excel file containing the data. Either click or drag the file into the drop zone from your file browser.

![Select spreadsheet](images/upload-selection.png)

After uploading a spreadsheet, the column corresponding to the sample name must be selected. After selecting the column header, press the `Preview the data` button.
After uploading a spreadsheet, you will be brought to the `Map Columns` step. The column corresponding to the sample name must be selected.

![Select name column.](images/upload-column.png)
![Select name column.](images/upload-column-before.png)

Rows that do not match an existing sample name are identified with the `New` tag. If selected, these samples will automatically be created. Rows that do match an existing sample name will be updated. Only select the rows that are to be uploaded and press the `Upload the data` button.
Once the sample name column is selected, a table will be displayed listing all the metadata fields. You can review the existing and target metadata field restrictions here. Press the `Review the data` button to continue.

![Select name column.](images/upload-column-after.png)

You may select the rows that are to be uploaded on the `Review Data` step. Rows that do not match an existing sample name are identified with the `New` tag. If selected, these samples will automatically be created. Rows that match an existing sample name will be updated.

![Preview Upload](images/upload-preview.png)

Rows that have an invalid sample name will be highlighted in red. These errors should be fixed within the spreadsheet and re-imported.
Rows that have an invalid sample name will be highlighted in red. These errors should be fixed within the spreadsheet and re-imported. Press the `Upload the data` button to continue.

![Upload Preview Errors](images/upload-preview-errors.png)

The complete page will be displayed on a successful upload. Clicking on the `Upload another file` button will redirect to the beginning of a new upload.
Progress will be displayed while uploading. Please be patient while uploading large data sets. Do not close the window or leave the page.

![Upload Preview Errors](images/loading.png)

On a successful upload, you will be brought to the `Complete` step. Clicking on the `Upload another file` button will redirect to the beginning of a new upload.

![Upload Preview Success](images/upload-preview-success.png)

Expand Down
Original file line number Diff line number Diff line change
@@ -1,19 +1,22 @@
package ca.corefacility.bioinformatics.irida.model.sample;

import ca.corefacility.bioinformatics.irida.model.sample.metadata.MetadataEntry;
import org.hibernate.envers.Audited;
import org.springframework.data.jpa.domain.support.AuditingEntityListener;
import java.util.List;
import java.util.Objects;

import javax.persistence.*;
import javax.validation.constraints.NotNull;
import java.util.List;
import java.util.Objects;

import org.hibernate.envers.Audited;
import org.springframework.data.jpa.domain.support.AuditingEntityListener;

import ca.corefacility.bioinformatics.irida.model.sample.metadata.MetadataEntry;

/**
* Describes an individual field in a {@link MetadataTemplate}.
*/
@Entity
@Table(name = "metadata_field")
@Table(name = "metadata_field",
uniqueConstraints = @UniqueConstraint(columnNames = { "label" }, name = "UK_METADATA_FIELD_LABEL"))
@Audited
@EntityListeners(AuditingEntityListener.class)
public class MetadataTemplateField {
Expand Down

This file was deleted.

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
package ca.corefacility.bioinformatics.irida.ria.web.ajax.dto;

/**
* UI response to create a new sample
*/
public class CreateSampleResponse extends UpdateSampleResponse {
private Long sampleId;

public CreateSampleResponse(String errorMessage) {
super(true, errorMessage);
}

public CreateSampleResponse(Long sampleId) {
super(false);
this.sampleId = sampleId;
}

public Long getSampleId() {
return sampleId;
}

public void setSampleId(Long sampleId) {
this.sampleId = sampleId;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,19 @@
* UI Request to update an existing sample
*/
public class UpdateSampleRequest extends CreateSampleRequest {
public UpdateSampleRequest(String name, String organism, String description, List<MetadataEntryModel> metadata) {
private Long sampleId;

public UpdateSampleRequest(Long sampleID, String name, String organism, String description,
List<MetadataEntryModel> metadata) {
super(name, organism, description, metadata);
this.sampleId = sampleID;
}

public Long getSampleId() {
return sampleId;
}

public void setSampleId(Long sampleId) {
this.sampleId = sampleId;
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
package ca.corefacility.bioinformatics.irida.ria.web.ajax.dto;

/**
* UI response to update an existing sample
*/
public class UpdateSampleResponse {
private boolean error;
private String errorMessage;

public UpdateSampleResponse() {
}

public UpdateSampleResponse(boolean error) {
this.error = error;
}

public UpdateSampleResponse(boolean error, String errorMessage) {
this.error = error;
this.errorMessage = errorMessage;
}

public boolean isError() {
return error;
}

public void setError(boolean error) {
this.error = error;
}

public String getErrorMessage() {
return errorMessage;
}

public void setErrorMessage(String errorMessage) {
this.errorMessage = errorMessage;
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
package ca.corefacility.bioinformatics.irida.ria.web.ajax.dto.ajax;

import java.util.Map;

/**
* AJAX response to return multiple responses to the client
*/
public class AjaxMultipleResponse extends AjaxResponse {
private Map<String, Object> responses;

public AjaxMultipleResponse(Map<String, Object> responses) {
this.responses = responses;
}

public Map<String, Object> getResponses() {
return responses;
}
}

Original file line number Diff line number Diff line change
Expand Up @@ -115,8 +115,12 @@ public List<ProjectMetadataField> getMetadataFieldsForProject(@RequestParam Long
@PostMapping("/fields")
public ResponseEntity<AjaxResponse> createMetadataFieldsForProject(@RequestParam Long projectId,
@RequestBody List<ProjectMetadataField> fields, Locale locale) {
return ResponseEntity.ok(
new AjaxSuccessResponse(service.createMetadataFieldsForProject(projectId, fields, locale)));
try {
return ResponseEntity.ok(
new AjaxSuccessResponse(service.createMetadataFieldsForProject(projectId, fields, locale)));
} catch (Exception e) {
return ResponseEntity.status(HttpStatus.CONFLICT).body(new AjaxErrorResponse(e.getMessage()));
}
}

/**
Expand Down
Loading