Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

adding performance results #140

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

adding performance results #140

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 33 additions & 10 deletions performance-testing/results/results.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,15 +53,6 @@ Uplift of peak load by 25%

# Throughput #

## Peak 1 Hour ##

| Modality | Transactions | Model executions |
| ---------- | ------------ | ---------------- |
| X-ray | 120 | 120 |
| Ultrasound | 50 | 5 |
| CT | 30 | 21 |
| MRI | 25 | 17.5 |

## Avg 1 Hour ##

| Modality | Transactions | Model executions |
Expand All @@ -70,6 +61,14 @@ Uplift of peak load by 25%
| Ultrasound | 28 | 2.8 |
| CT | 10 | 7 |
| MRI | 13 | 9.1 |
## Peak 1 Hour ##

| Modality | Transactions | Model executions |
| ---------- | ------------ | ---------------- |
| X-ray | 120 | 120 |
| Ultrasound | 50 | 5 |
| CT | 30 | 21 |
| MRI | 25 | 17.5 |

## Stress 1 Hour ##

Expand Down Expand Up @@ -135,16 +134,40 @@ Send through the same study 5 times, with a 90 second gap to get average metric

### Baseline 2 ###
#### Description ####
Retest of the MIG following a change to how it was saving data to MinIO.
Retest of the MIG following a change to how it saved data to MinIO which resulted in ~ 50% reduction in time.

#### Metrics ####

| | DICOM Payload Processed | DICOM Payload Processed |
| -------- | ----------------------- | ----------------------- |
| Modality | Average (sec) | Max (sec) |
| CT | 14 | 16.2 |

### Load (Avg) ###

#### Description ####
To evaluate the performance of MONAI Deploy under typical or average usage conditions for a large Trust. Please see [Avg 1 Hour](#avg-1-hour) for details of volumetrics
#### Metrics ####
| DICOM Payload Processed | DICOM Payload Processed | Task Dispatched | Task Dispatched | Task Created | Task Created | Task Updated | Task Updated |
| ----------------------- | ----------------------- | --------------- | --------------- | ------------- | ------------ | ------------- | ------------ |
| Average (sec) | Max (sec) | Average (sec) | Max (sec) | Average (sec) | Max (sec) | Average (sec) | Max (sec) |
| 5.8 | 8.8 | 0.8 | 1.3 | 1.9 | 26.4 | \- | \- |

** DICOM Payload Processed was only tested with low instance studies. Bug to be raised which ensures that the async nature of the sending of DICOMs doesn't timeout after 60 seconds.

** Task updated metrics were not useful at high load as systems are very chatty, work needs to be done to differentiate useful messages better.

#### Failure/ Successful transactions ####

| | Transactions | Errors | Details |
| ----------------- | ------------ | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Payload processed | 101 | 0 | |
| Task Dispatched | 88 | 52 | WorkflowRequestRequeuePayloadProcessError was thrown 52 times for 14 payloads, this was due to the way the WorkflowManager was processing "Unable to locate a matching workflow for the given workflow request". It would seem that the WorkflowManager requeues WorkflowRequetEvents that do not match a Workflow AET and then throws an exception because that payload is already received and saved in Mongo. This does not affect functionality or performance |
| Task Created | 88 | 0 | |
| Task Updated | \- | \- | |
| Small-app | 62 | 0 | |
| Medium-app | 14 | 0 | |
| Large-app | 12 | 0 |

### Load (Peak) ###
#### Description ####
Expand Down
Binary file added performance-testing/static/Pre_Prod_Avg_Nvidia.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added performance-testing/static/Pre_Prod_Avg_Overview.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.