Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GalaxyBenchmarker V2 for storage benchmarking #2

Open
wants to merge 222 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
222 commits
Select commit Hold shift + click to select a range
007b774
Make galaxy_benchmarker a module
smoehrle Jan 10, 2022
573c601
Move bridges/connectors into seperate module
smoehrle Jan 10, 2022
37a33bc
Move bioblend/galaxy logic into galaxy bridge
smoehrle Jan 10, 2022
ce83307
Move data models into models-module
smoehrle Jan 10, 2022
0b908ba
Refactor task: remove dependency on benchmark
smoehrle Jan 10, 2022
4289962
Refactor: Move condor and galaxy specific logic from metrics.py to we…
smoehrle Jan 18, 2022
da0ec55
Refactor: Move function run_galaxy_benchmark into class
smoehrle Jan 18, 2022
f81d290
Refactor: Extact first benchmark cold/warm
smoehrle Jan 19, 2022
d6848d0
Refactor: extract config, new benchmark generation
smoehrle Jan 19, 2022
36fb0de
Refactor: use module names as logger
smoehrle Jan 24, 2022
83e9694
Refactor: Move run logik into benchmarker
smoehrle Jan 24, 2022
6cac6a1
Refactor: change benchmark config from list to dict
smoehrle Jan 24, 2022
d9caaee
Refactor: remove dead code
smoehrle Jan 24, 2022
0266a15
Refactor: tasks
smoehrle Jan 24, 2022
2822085
Refactor: tasks, reintegrat into benchmark
smoehrle Jan 24, 2022
d684f8f
Refactor: Add AnsibleDestination for AnsibleTasks
smoehrle Jan 24, 2022
4e53486
Refactor: rename runs_per_workflow to repetitions
smoehrle Jan 24, 2022
a4d8460
First draft of posix benchmarks
smoehrle Jan 24, 2022
a8d14e6
Remove concept of configurable pre/post tasks
smoehrle Jan 25, 2022
f8b395a
Make AnsibleDestination user optional
smoehrle Jan 25, 2022
0001fdc
Add posix_setup_time_benchmark
smoehrle Jan 25, 2022
acb44e2
Probe influxdb connection
smoehrle Jan 25, 2022
e4684be
Improve result output for influxdb and logfile
smoehrle Jan 25, 2022
4b85217
Posix fio benchmark is working, result evaluation still missing
smoehrle Jan 25, 2022
7abd907
Posix fio benchmark: fully working
smoehrle Jan 26, 2022
1a12055
Fix docker-compose setup; Add SSH-Agent forwarding
smoehrle Jan 26, 2022
02d9bcf
Move to pyproject.toml
smoehrle Jan 26, 2022
33b3266
Add isort for import sorting
smoehrle Jan 26, 2022
f080e70
Add black for code formatting
smoehrle Jan 26, 2022
fac3fd6
Better seperation between __init__ and from_config constructors for A…
smoehrle Jan 26, 2022
9d6ec48
Fix package installation/code mounting for docker setup
smoehrle Jan 26, 2022
5427e8c
Add multiple scenarios for FIO-Benchmark
smoehrle Jan 31, 2022
bd1c9f6
Automated (un)install of containerized iRODS as ansible script
smoehrle Feb 1, 2022
bf45073
Extract fio config into dataclass
smoehrle Feb 2, 2022
77e514a
Add new benchmark: PosixFioIopsOverTime
smoehrle Feb 2, 2022
e3e744b
Add Read-Write Benchmarks
smoehrle Feb 7, 2022
a4628b4
Move save_results_to_file into from Benchmarker to Benchmark
smoehrle Feb 7, 2022
444413d
Add barrier.py which syncs fio jobs
smoehrle Feb 7, 2022
c2dbd2a
Add signal handling and `make stop` to stop running benchmark
smoehrle Feb 8, 2022
7ee8152
Setup irods only on the first startup
smoehrle Feb 8, 2022
facc19c
New benchmark: Fio: Throughput over different filesizes
smoehrle Feb 9, 2022
a4e8842
Automate iRODS setup and configuration
smoehrle Feb 14, 2022
32dce17
Re-work config, add 01_verify_setup.yml example
smoehrle Feb 15, 2022
c80eb03
Add goal config for example 03
smoehrle Feb 15, 2022
23f3f48
Add basic structure for example 03
smoehrle Feb 15, 2022
6f1ac1f
example03 is parsable
smoehrle Feb 15, 2022
650448c
Automate irods-fuse setup and cleanup
smoehrle Feb 15, 2022
24bb5e4
Add example with available options
smoehrle Feb 15, 2022
04746c4
Cleanup, fix some typhint issues
smoehrle Feb 15, 2022
7c535d5
Un-hardcode IP for irods server
smoehrle Feb 15, 2022
2a80c03
Archive all files from prev project currently not needed
smoehrle Feb 15, 2022
1d9f36e
Cleanup playbooks
smoehrle Feb 15, 2022
008d886
Add davrods server
smoehrle Feb 15, 2022
461a185
Fix davrods SSL-issue
smoehrle Feb 28, 2022
1c9ae76
Add webdav client
smoehrle Feb 28, 2022
65e3522
Fix webdav client permissions
smoehrle Feb 28, 2022
dc86629
Fix ansible tasks with certain configuration options
smoehrle Mar 1, 2022
930dd13
Add checks for docker install
smoehrle Mar 1, 2022
7a4bfd4
Add containerized fio for comparison
smoehrle Mar 1, 2022
3dcf5ad
Add some type hints
smoehrle Mar 2, 2022
cc0375d
Rename playbooks/ -> ansible/
smoehrle Mar 2, 2022
4964e37
Remove AnsibleDestination, use ansible inventory instead
smoehrle Mar 2, 2022
e054fe7
Add sample inventory
smoehrle Mar 2, 2022
4d6e709
Add some readme doc and improve usage
smoehrle Mar 2, 2022
603ce41
Fix issues with changing to ansible inventory
smoehrle Mar 15, 2022
784486c
Make fio refill_buffers and time_based configurable
smoehrle Mar 15, 2022
13028fc
Split install irods task
smoehrle Mar 15, 2022
3c00dde
Add irods with S3 Backend
smoehrle Mar 15, 2022
2370019
Cleanup and fix containerized fio
smoehrle Mar 16, 2022
07b5aee
Add dd as tool
smoehrle Mar 16, 2022
430e49e
Restructure fio benchmarks
smoehrle Mar 22, 2022
4b1d021
wip
smoehrle Mar 22, 2022
1129f53
Remove unused influxdb
smoehrle Mar 22, 2022
ea04b08
Add more tags to results
smoehrle Mar 22, 2022
1865279
Fix linter errors
smoehrle Mar 22, 2022
f8c3388
remove barrier for fio
smoehrle Mar 23, 2022
fb2d71e
Uniform result file format
smoehrle Mar 23, 2022
2e3e733
Add troughput evaluation
smoehrle Apr 11, 2022
d1c3247
Experiment 2: write throughput dd vs fio
smoehrle Apr 12, 2022
0f8f6df
experiment 3: containerized vs non-containerized fio
smoehrle Apr 12, 2022
a555964
Experiment 5: netapp cold/warm
smoehrle Apr 12, 2022
cba047f
Experiment 4: daytime evaluation
smoehrle Apr 12, 2022
c05e25d
Simplify base benchmark and destination handling
smoehrle Apr 18, 2022
2c6558b
Run post task even when pre-task ails
smoehrle Apr 18, 2022
1e1de23
Fix user permissions
smoehrle Apr 19, 2022
fee75f4
Add flags which control the execution
smoehrle Apr 19, 2022
fb9727d
Update lock-file
smoehrle Apr 19, 2022
71f121d
Fix davrods and fuse
smoehrle Apr 19, 2022
eac82b2
Special case handling for fuse and fio read benchmark
smoehrle Apr 20, 2022
a0361ba
Improve fio stability
smoehrle Apr 25, 2022
b97a891
Add mdtest
smoehrle Apr 25, 2022
1c2c318
Rename posix benchmark
smoehrle Apr 25, 2022
99b9827
Small improvements
smoehrle Apr 25, 2022
ce7052f
Add S3 benchmark
smoehrle May 9, 2022
9cbe559
Add make target for cronjob
smoehrle May 9, 2022
495794c
Update daytime_evaluation
smoehrle May 9, 2022
070ddde
Add optional raw results
smoehrle May 16, 2022
08cebfa
More fio benchmarks, add ramp-time for fio
smoehrle May 16, 2022
43452bd
Reorganize examples
smoehrle May 23, 2022
a74ba56
Add cli arg for filtering benchmakrs
smoehrle May 23, 2022
5aac884
Update irodsfs version
smoehrle May 23, 2022
b8a5fe6
Install specific version of irodsfs
smoehrle May 23, 2022
04101c6
update dd
smoehrle May 23, 2022
1c5495f
Fix example 02, add variable ioengine for fio
smoehrle May 24, 2022
41c1f15
Temp fix
smoehrle May 24, 2022
8526b79
more dd vs fio
smoehrle May 25, 2022
f813556
Clear up MB vs MiB
smoehrle May 30, 2022
2ef01ae
Unify benchmark config
smoehrle May 30, 2022
9b8f54f
Remove code duplication
smoehrle May 30, 2022
4c2a9dc
Remove more code duplication
smoehrle May 30, 2022
4027659
Remove unnecessary runs
smoehrle May 30, 2022
f24f69f
Improve experiments
smoehrle May 31, 2022
1042615
Current state of eval jupiter notebook
smoehrle May 31, 2022
910e5f5
Rename some exmaples
smoehrle Jun 6, 2022
cfbe63a
Pin mdtest installation
smoehrle Jun 7, 2022
9e3bddf
Fix bug in posix benchmark
smoehrle Jun 8, 2022
31a1420
Some cleanup
smoehrle Jun 28, 2022
c498bb9
Add tool warp
smoehrle Jun 28, 2022
1502836
Allow multiple config files as input
smoehrle Jul 4, 2022
f74f748
Add BenchmarkCompare for comparison of benchmark
smoehrle Jul 4, 2022
ec6f9fb
Add parallel dd
smoehrle Jul 5, 2022
74fb3d4
Change dd parallel 2->4
smoehrle Jul 5, 2022
287dc36
add warp result parsing
smoehrle Jul 5, 2022
0c0ccb0
Add logging after run has finsihed
smoehrle Jul 5, 2022
b5ffb94
Prepare cronjob
smoehrle Jul 5, 2022
592e5a9
Fix rounding error message
smoehrle Jul 5, 2022
31f8be9
Fix os.rename error
smoehrle Jul 5, 2022
a3d51f0
Fix path in docker-compose
smoehrle Jul 5, 2022
83a9232
Fix dd evaluation logic
smoehrle Jul 12, 2022
037ae3b
Plots and results
smoehrle Jul 12, 2022
048da3b
Ignore png's
smoehrle Jul 12, 2022
cf01928
Move some examples
smoehrle Jul 12, 2022
7075d96
Upgrade irods to 4.3.0, pin versions
smoehrle Jul 12, 2022
e3eab1a
Prepare second experiment runs
smoehrle Jul 12, 2022
d4dfa3b
Improve config parsing
smoehrle Jul 12, 2022
d537007
Move result parsing after saving raw results
smoehrle Jul 12, 2022
4955028
Use copy for raw results
smoehrle Jul 12, 2022
7643d65
Fix warp result parsing
smoehrle Jul 12, 2022
e514595
Fix s3 config
smoehrle Jul 12, 2022
94ab701
Move examples around
smoehrle Jul 13, 2022
9160e7f
Add param eval for s3 tools
smoehrle Jul 13, 2022
488cb9f
Preprare s3bench vs warp
smoehrle Jul 13, 2022
4ff841d
More cleanup in examples
smoehrle Jul 13, 2022
e5d6c5a
Galaxy server deployment part1
smoehrle Jul 13, 2022
022a4b1
Undo version upgrade of irods (befor: 4.3.0, after: 4.2.10)
smoehrle Jul 18, 2022
dab2b0c
Update fio to latest version 3.30
smoehrle Jul 18, 2022
4ad4777
Add s3 to daytime eval
smoehrle Jul 18, 2022
3145b0d
More results and evaluation
smoehrle Jul 25, 2022
f5dd43b
Add some values to param eval
smoehrle Jul 25, 2022
5a091ad
Next round of experiments
smoehrle Jul 25, 2022
b4bb95f
Make fio benchmark more robust
smoehrle Jul 25, 2022
a7833a9
Rename utils/posix.py
smoehrle Jul 25, 2022
39d55cc
Distinguish between PosixDestination and AnsibleDestination
smoehrle Jul 25, 2022
3821475
Draft: Setup galaxy and trigger job
smoehrle Jul 26, 2022
aab9747
Config galaxy server with mount backend
smoehrle Jul 27, 2022
6690cce
Config galaxy server with s3 backend
smoehrle Jul 27, 2022
8f0a081
Check s3 content and remove files afterwards
smoehrle Aug 1, 2022
0d7f387
Format code
smoehrle Aug 1, 2022
be7ce28
Move s3 details into config
smoehrle Aug 1, 2022
f13b30f
Unhardcode credentials (creds also rotated)
smoehrle Aug 1, 2022
629e268
Fix galaxy with davrods backend
smoehrle Aug 1, 2022
f2be58f
Unify galaxy job input for 06_benchmark_galaxy
smoehrle Aug 1, 2022
1c1b38a
Add direct galaxy->irods integration benchamrk
smoehrle Aug 1, 2022
f341dcb
First galaxy benchamrks
smoehrle Aug 1, 2022
857093c
Remove old/unused files/code
smoehrle Aug 2, 2022
d011b07
Fix cleanup for galaxy
smoehrle Aug 2, 2022
30c614e
Add missing file
smoehrle Aug 2, 2022
3572052
Remove more dead code
smoehrle Aug 2, 2022
a60ae20
Bugfixes after refactor
smoehrle Aug 2, 2022
62b3687
Add galaxy test cases
smoehrle Aug 2, 2022
dc30cf0
Improve timeouts, ensure an empty bucket
smoehrle Aug 2, 2022
fe49ed8
check items for size as well
smoehrle Aug 2, 2022
1f7b666
Fix name for variable
smoehrle Aug 2, 2022
c03cee6
Bugfix for s3 verification
smoehrle Aug 2, 2022
742cdc0
More bugfixes
smoehrle Aug 2, 2022
bfa07a4
Bugfix
smoehrle Aug 2, 2022
6374757
Disable s3 for large files
smoehrle Aug 2, 2022
f3632b0
Add 10G test
smoehrle Aug 2, 2022
4eeaf8e
More bugfixes
smoehrle Aug 2, 2022
2ec42c0
Increase timeout
smoehrle Aug 2, 2022
7848c0b
Change names of benchmarks
smoehrle Aug 3, 2022
011e5c5
Merge setup_galaxy_server.yml and setup_galaxy_server_s3.yml
smoehrle Aug 3, 2022
a7c895b
Merge setup_irods_server.yml and setup_irods_server_with_davrods.yml
smoehrle Aug 3, 2022
f060e79
Update examples/05*
smoehrle Aug 3, 2022
fe2fbcd
Merge cleanup_irods_server and cleanup_irods_server_with_davrods
smoehrle Aug 3, 2022
59b7eec
Enable davrods on s3
smoehrle Aug 3, 2022
6874cda
Add galaxy_via_davrods_on_s3
smoehrle Aug 3, 2022
a7e5881
update acitve benchmarks
smoehrle Aug 3, 2022
9dae63f
More longterm
smoehrle Aug 8, 2022
e2337c9
Update plot code
smoehrle Aug 16, 2022
e456ad3
Update s3-benchmark results
smoehrle Aug 16, 2022
f2ff3c4
Bugfixes
smoehrle Aug 16, 2022
851f9d7
Make fio more robust
smoehrle Aug 16, 2022
d77bc2d
Add local benchmark
smoehrle Aug 16, 2022
61d6cb6
Update docker-compose
smoehrle Aug 16, 2022
d64b46e
Add flag for fio: preare_run_benchmark_in_tmp
smoehrle Aug 17, 2022
54f187a
Remove owner check
smoehrle Aug 17, 2022
93f4180
Fix path
smoehrle Aug 17, 2022
00dfc3a
Update s3 benchmark
smoehrle Aug 22, 2022
16dc652
Tweaks on experiments
smoehrle Aug 22, 2022
da6c43f
Fix s3benchmakr result parsing
smoehrle Aug 22, 2022
c8cbcfc
Add check for S3 keys
smoehrle Aug 23, 2022
cf4b64b
Fix mdtest config
smoehrle Aug 23, 2022
13c1269
Fix
smoehrle Aug 23, 2022
cb42bd2
Fix names of metadata benchamrk
smoehrle Aug 23, 2022
bdfb0d3
Add mdtest param study
smoehrle Aug 23, 2022
09a3f63
More parameval mdtest
smoehrle Aug 23, 2022
1504201
Add netapp to mdtest
smoehrle Aug 23, 2022
4a3c830
Update mdtest benchmarks
smoehrle Aug 23, 2022
2e954fc
Update docker-compose
smoehrle Aug 23, 2022
7c40ee5
Add flush to mdtest
smoehrle Aug 23, 2022
8307b13
Add results and current evaluation script
smoehrle Aug 29, 2022
e9483f7
Fix 10x1G benchamrks
smoehrle Aug 30, 2022
b978020
Remove ssd tank from galaxy benchamrks
smoehrle Aug 30, 2022
cf51edc
Add benchamrsk for glx_netapp
smoehrle Aug 30, 2022
db7071d
Update docker-compose
smoehrle Aug 30, 2022
5f8ce8f
Add missing parameter
smoehrle Aug 30, 2022
4344b7a
More galaxy resutls
smoehrle Aug 31, 2022
5b7ff6c
Better galaxy restart in case of race-contition prevents galaxy to bo…
smoehrle Aug 31, 2022
0f45c0d
Add some documentation
smoehrle Aug 31, 2022
cdf8aff
Latest evaluation
smoehrle Sep 5, 2022
f1b0f04
Add link to result files
smoehrle Sep 5, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# GalaxyBenchmarker generated files
logs/
results/

# Python stuff
venv/
.venv/
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ tools.yml
*.html
results/*
tool_test_output.*
.vscode
evaluation/*.png

### Autogenerated: ###
# Byte-compiled / optimized / DLL files
Expand Down
32 changes: 26 additions & 6 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,8 +1,28 @@
FROM python:3.7
FROM python:3.10

COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt
RUN pip uninstall bioblend -y
RUN git clone https://github.com/galaxyproject/bioblend.git && cd bioblend && python setup.py install
ENV PIP_NO_CACHE_DIR=1 \
PIP_DISABLE_PIP_VERSION_CHECK=1

RUN cd / && git clone https://github.com/usegalaxy-eu/workflow-testing.git
RUN \
apt-get update \
&& apt-get install -y \
rsync \
tini \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/* /tmp/*

RUN pip install poetry

WORKDIR /src
COPY poetry.lock pyproject.toml /src/

RUN poetry config virtualenvs.create false \
&& poetry install --no-dev --no-interaction --no-ansi

RUN git clone https://github.com/usegalaxy-eu/workflow-testing.git /workflow-testing

COPY . /src/

COPY scripts/entrypoint.sh /entrypoint.sh

ENTRYPOINT ["/usr/bin/tini", "-v", "--", "/entrypoint.sh"]
36 changes: 36 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
run:
docker-compose up -d
docker-compose logs -f benchmarker

run_sudo:
sudo -E docker-compose up -d
sudo docker-compose logs -f benchmarker

.PHONY: run_cron
run_cron:
/usr/local/bin/docker-compose up -d

stop:
echo "Gracefull shutdown. Benchmarker has 30 seconds to save results"
docker-compose stop -t 30 benchmarker

stop_sudo:
echo "Gracefull shutdown. Benchmarker has 30 seconds to save results"
sudo docker-compose stop -t 30 benchmarker

build:
docker-compose build

build_sudo:
sudo docker-compose build

lint: lint_isort lint_black

lint_isort:
poetry run isort galaxy_benchmarker

lint_black:
poetry run black --target-version py310 galaxy_benchmarker

lint_mypy:
poetry run mypy galaxy_benchmarker
Loading