-
Notifications
You must be signed in to change notification settings - Fork 52
/
README
804 lines (705 loc) · 45.4 KB
/
README
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
Block Storage Benchmark
This benchmark suite uses fio, scripts for automation and graph/PDF
generators for reporting to replicate the SNIA Solid State Storage
(SSS) Performance Test Specification (PTS) Enterprise v1.1. This specification
includes 8 test each measuring different block storage performance
characteristics. The specification is available here:
http://snia.org/sites/default/files/SSS_PTS_Enterprise_v1.1.pdf
This benchmark suite uses gnuplot and wkhtmltopdf to generate test reports
based on the SNIA specification. An example report is available here:
https://github.com/cloudharmony/block-storage/blob/master/reports/sample-report.pdf?raw=true
A few notes on cloud/virtualization limitations as they pertain to SNIA test
specifications:
1. Due to virtualization, direct access to hardware commands like ATA secure
erase are generally unavailable in cloud environments. Thus, the recommended
Purge methods prescribed by the test specification are generally not
feasible in a cloud environment. As a workaround, where supported, full
device TRIM is applied prior to preconditioning and testing cycles (using
`blkdiscard` for devices and `fstrim` for mounted volumes). If ATA secure
erase and TRIM are not supported, devices are zero filled as a last resort.
The purge method used is documented in test output. If test targets are
rotational devices, zero fill formatting is used instead. Thus, the purge
order of precedence is: 1) ATA secure erase; 2) TRIM (SSD only) 3) zero
fill. These can be overridden using the nosecureerase, notrim and nozerofill
test parameters.
2. The test specification requirement for write caches to be disabled cannot
be guaranteed in a cloud environment.
3. Often virtual machine block storage devices are partial logical volumes
on a physical drive that may contain additional logical volumes in use by
other users. Because of this, testing may not be across the full LBA of
the physical drive (i.e. ActiveRange < 100%). Additionally, the possible
shared state of the physical drive may contribute to higher variability
and difficulty or inability to achieve the prescribed +/-10% condition for
Steady State verification. This may also restrict the ability to
consistently reproduce results across multiple tests
TESTING PARAMETERS
The SNIA test specification allow for some user configurable parameters. These
are described below. To start testing, use `run.php`. For runtime options, use
`run.php --help`. Each of the parameters below may be specified using CLI
arguments (e.g. `run.php --test=iops`) or `bm_param_` prefixed environment
variables (e.g. `export bm_param_test=iops`)
* active_range LBA range to use for testing represented as a
percentage. Default is 100, or 100% per the SNIA
test specification. To test across a smaller
range, this parameter may be reduced. If the
test targets are not devices, test size is total
free space - 100 MB. WARNING: if multiple volume
type targets are specified, active range will be
limited to the least amount of free space on all
targets
* collectd_rrd If set, collectd rrd stats will be captured from
--collectd_rrd_dir. To do so, when testing starts,
existing directories in --collectd_rrd_dir will
be renamed to .bak, and upon test completion
any directories not ending in .bak will be zipped
and saved along with other test artifacts (as
collectd-rrd.zip). User MUST have sudo privileges
to use this option
* collectd_rrd_dir Location where collectd rrd files are stored -
default is /var/lib/collectd/rrd
* fio Optional explicit path to the fio command -
otherwise fio in PATH will be used
* fio_* Optional fio runtime parameters. By default, fio
parameters are generated by the test script per
the SNIA test specification. Use this parameter to
override default fio settings
(e.g. fio_ioengine=sync)
* font_size The base font size pt to use in reports and graphs.
All text will be relative to this size (i.e.
smaller, larger). Default is 9. Graphs use this
value + 4 (i.e. default 13). Open Sans is included
with this software. To change this, simply replace
the reports/font.ttf file with your desired font
* highcharts_js_url URL to highcharts.js. Highcharts is used to render
3D charts in reports. Use 'no3dcharts' to disable
3D charts. Default for this parameter is
http://code.highcharts.com/highcharts.js
* highcharts3d_js_url URL to highcharts-3d.js. Highcharts is used to
render 3D charts in reports. Use 'no3dcharts' to
disable 3D charts. Default for this parameter is
http://code.highcharts.com/highcharts-3d.js
* jquery_url URL for jquery. jquery is used by Highcharts.
Highcharts is used to render 3D charts in reports.
Use 'no3dcharts' to disable 3D charts. Default for
this parameter is
http://code.jquery.com/jquery-2.1.0.min.js
* meta_compute_service The name of the compute service this test pertains
to. Used for report headers. May also be specified
using the environment variable bm_compute_service
* meta_compute_service_id The id of the compute service this test pertains
to. Added to saved results. May also be specified
using the environment variable
bm_compute_service_id
* meta_cpu CPU descriptor to use for report headers. If not
specified, it will be set using the 'model name'
attribute in /proc/cpuinfo
* meta_drive_interface Optional drive interface descriptor to use for
report headers (e.g. SATA 6Gb/s)
* meta_drive_model Optional drive model descriptor to use for report
headers (e.g. Intel DC S3700)
* meta_drive_type Optional drive type descriptor to use for report
headers (e.g. SATA, SSD, PCIe)
* meta_instance_id The compute service instance type this test pertains
to (e.g. c3.xlarge). Used for report headers
* meta_memory Memory descriptor to use for report headers. If not
specified, the amount of memory will be used (as
reported by 'free')
* meta_notes_storage Optional notes to display in the Storage Platform
header column
* meta_notes_test Optional notes to display in the Test Platform
header column
* meta_os Operating system descriptor to use for report
headers. If not specified, this meta data will be
derived from the first line of /etc/issue
* meta_provider The name of the cloud provider this test pertains
to. Used for report headers. May also be specified
using the environment variable bm_provider
* meta_provider_id The id of the cloud provider this test pertains
to. Added to saved results. May also be specified
using the environment variable
bm_provider_id
* meta_region The compute service region this test pertains to.
Used for report headers. May also be specified
using the environment variable bm_region
* meta_resource_id An optional benchmark resource identifiers. Added
to saved results. May also be specified using the
environment variable bm_resource_id
* meta_run_id An optional benchmark run identifiers. Added to
saved results. May also be specified using the
environment variable bm_run_id
* meta_storage_config Storage configuration descriptor to use for report
headers. If not specified, 'Unknown' will be
displayed in this column
* meta_storage_vol_info If testing is against a volume, this optioanl
parameter may be used to designate setup of that
volume (e.g. Raid, File System, etc.). Only
displayed when targets are volumes. If this
parameter is not specified the file system type
for each target volume will be included in this
column
* meta_test_id Identifier for the test. Used for report headers.
If not specified, this header column will be blank
* meta_test_sw Name/version of the test software. Used for report
headers. If not specified, this header column will
be blank
* no3dcharts Don't generate 3D charts. Unlike 2D charts rendered
with the free chart utility gnuplot, 3D charts are
rendered using highcharts - a commercial javascript
charting tool. highcharts is available for free for
non-commercial and development use, and for a
nominal fee otherwise. See http://www.highcharts.com
for additional licensing information
* nojson Don't generate JSON result or fio output files
* nopdfreport Don't generate PDF version of test report -
report.pdf. (wkhtmltopdf dependency removed if
specified)
* noprecondition Don't perform the default 2X 128K sequential
workload independent preconditioning (per the SNIA
test specification). This step precedes workload
dependent preconditioning
* nopurge Don't require a purge for testing. If this
parameter is not set, and at least 1 target could
not be purged, testing will abort. This parameter
is implicit if --nosecureerase, --notrim and
--nozerofill are all specified
* norandom Don't test using random (less compressible) data.
Use of random data for IO is a requirement of the
SNIA test specification
* randommap Random maps are allocated at init time to track
written to blocks and duplicate block writes. When
used, random maps must be allocated in memory at
init time. The memory allocation for these can be
problematic for large test volume (e.g. 16TB volume
= 4.2GB memory). If this option is not set, random
fio tests will be executed using the --norandommap
and --randrepeat=0 fio options
* noreport Don't generate html or PDF test reports -
report.zip and report.pdf (gnuplot, wkhtmltopdf and
zip dependencies removed if specified)
* nosecureerase Don't attempt to secure erase device targets prior
to test cycles (this is the first choice - hdparm
dependency removed if specified). This parameter is
implicit if --secureerase_pswd is not provided
* notrim Don't attempt to TRIM devices/volumes prior to
testing cycles (util-linux dependency removed if
specified)
* nozerofill Don't zero fill rotational devices (or SSD devices
when TRIM is not supported) prior to testing
cycles. Zero fill applies only to device targets
* oio_per_thread The outstanding IO per thread (a.k.a. queue depth).
This translates to the fio `iodepth` parameter.
Total outstanding IO for a given test is
`threads` * `threads_oio`. Per the SNIA test
specification, this is a user definable parameter.
For latency tests, this parameter is a fixed value
of 1. Default value for this parameter is 64
* output The output directory to use for writing test artifacts
(JSON and reports). If not specified, the current
working directory will be used
* precondition_passes Number of passes for workload independent
preconditioning. Per the SNIA test specification
the default is 2X. Use this or the --noprecondition
argument to change this default behavior
* savefio Include results from every fio test job in save
output
* secureerase_pswd In order for ATA secure erase to be attempted for
device purge (prior to test invocation), you must
first set a security password using the command:
sudo hdparm --user-master u --security-set-pass [pswd] /dev/[device]
The password used should be supplied using this
test parameter. If it is not supplied, ATA secure
erase will not be attempted. If this parameter is
not specified, the hdparm dependent will be removed
* sequential_only If set, all random tests will be executed using
sequential IO instead
* skip_blocksize block sizes to skip during testing. This argument
may be repeated. Valid options are:
1m, 128k, 64k, 32k, 16k, 8k, 512b
* skip_workload workloads to skip during testing. This argument may
be repeated. Valid options are:
100/0, 95/5, 65/35, 50/50, 35/65, 5/95
* ss_max_rounds The maximum number of test cycle iterations to
allow for steady state verification. Default is
x=25 (per the SNIA test specification). If steady
state cannot be reached within this number of test
cycles (per the ss_verification ratio), testing
will terminate, and results will be designated as
having not achieved steady state. This parameter
may be used to increase (or decrease) the number of
test cycles. A minimum value of 5 is permitted
(the minimum number of cycles for the measurement
window)
* ss_verification Ratio to utilize for steady state verification. The
default is 10 or 10% per the SNIA test
specification. In order to achieve steady state
verification, the variance between the current test
cycle loop and the 4 that precede it cannot exceed
this value. In cloud environments with high IO
variability, it may be difficult to achieve the
default ratio and thus this value may be increased
using this parameter
* target REQUIRED: The target device or volume to use for
testing. This parameter may reference either the
physical device (e.g. /dev/sdc) or a mounted volume
(e.g. /ssd). TRIM will be attempted using
`blkdiscard` for a device and `fstrim` for a volume
if the targets are non-rotational. For rotational
devices, a zero fill will be used (i.e.
`dd if=/dev/zero`). Multiple targets may be
specified each separated by a comma. When multiple
targets are specified, the `threads` parameter
represents the number of threads per target (i.e.
total threads = # of targets * `threads`). Multiple
target tests provide aggregate metrics. With the
exception of latency tests, if multiple targets are
specified, they will be tested concurrently.
Sufficient permissions for the device/volume must
exist for the user that initiates testing.
WARNING: If a device is specified (e.g. /dev/sdc),
all data on that device will be erased during the
course of testing. All targets must be of the same
type (device or volume). WARNING: if multiple
volume type targets are specified, active range
will be limited to the least amount of free space
on all targets
* target_skip_not_present If set, targets specified that do not exist will be
ignored (so long as at least 1 target exists)
* test The SNIA SSS PTS tests to perform. One or more of
the following:
iops: IOPS Test - measures IOPS at a range
of random block sizes and read/write
mixes
throughput: Throughput Test - measures 128k and
1m sequential read and write
throughput (MB/s) in steady state
latency: Latency Test - measures IO response
times for 3 block sizes (0.5k, 4k and
8k), and 3 read/write mixes (100/0,
65/35 and 0/100). If multiple
`target` devices or volumes are
specified, latency tests are
performed sequentially
wsat: Write Saturation Test - measures how
drives respond to continuous 4k
random writes over time and total GB
written (TGBW). NOTE: this
implementation uses the alternate
steady state test method (1 minute SS
checks interspersed by 30 minute WSAT
test intervals)
hir: Host Idle Recovery - observes whether
the devices utilizes background
garbage collection wherein
performance increases with the
introduction of host idle times
between periods of 4k random writes
xsr: Cross Stimulus Recovery - tests how
the device handles transitions from
large block sequential writes to
small block random writes and back
NOT YET IMPLEMENTED
ecw: Enterprise Composite Workload -
measures performance in a mixed IO
environment
NOT YET IMPLEMENTED
dirth: Demand Intensity / Response Time
Histogram - measures performance
degradation when a device is subject
to a super saturating IO load
NOT YET IMPLEMENTED
Multiple tests may be specified each separated by
a comma. Default for this parameter is iops. Some
tests like 'hir' are specific to SSD type devices
* threads The number of threads to use for the test cycle.
Per the SNIA test specification, this is a user
definable parameter. The default value for this
parameter is the number of CPU cores. This
parameter may contain the token {cpus} which will
be replaced with the number of CPU cores present.
It may also contain a mathematical expression in
conjunction with {cpus} - e.g. {cpus}/2. If
`target` references multiple devices or volumes,
this parameter signifies the number of threads per
device. Thus total threads is # of targets *
`threads`. Latency tests are fixed at 1 thread.
This parameter is used to define the fio --numjobs
argument
* threads_per_core_max Max number of threads per CPU core - default is 2.
If this parameter causes the number of threads per
target to be less than 1, threads per target will
be increased to 1
* threads_per_target_max Max number of threads per target - default is 8
* timeout Max time to permit for testing in seconds. Default
is 24 hours (86400 seconds)
* trim_offset_end When invoking a blkdiscard TRIM, offset the lenth
by this number of bytes
* verbose Show verbose output - warning: this may produce a
lot of output
* wd_test_duration The test duration for workload dependent test
iterations in seconds. Default is 60 per the SNIA
test specification
* wd_sleep_between Optional sleep duration (seconds) to apply between
each workload dependent test. This may be a formula
containing the following dynamic values:
{duration} => duration of the last test (secs)
{size} => size of targets (GB)
{volumes} => number of target volumes
Ternary operations are also supported. This
parameter may be useful for credit based storage
platforms like Amazon EC2 EBS General Purpose
storage
(see: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html)
For example, per the Amazon EC2 EBS documentation
to ensure testing occurred within EBS burst
capacity, the following formula could be set for
this parameter:
--wd_sleep_between "{duration}*({size} >= 1000 ? 0 : ({size} >= 750 ? 0.33 : ({size} >= 500 ? 1 : ({size} >= 250 ? 3 : ({size} >= 100 ? 9 : 100)))))"
A shortcut to this formula is --wd_sleep_between ebs
DEPENDENCIES
This benchmark suite uses the following packages:
fio Performs actual testing - version 2.1+ required
gnuplot Generates graphs per the SNIA test specification. These graphs
are used in the PDF report
hdparm Used for ATA secure erase (when supported)
php-cli Test automation scripts (/usr/bin/php)
util-linux For TRIM operations using `blkdiscard` and `fstrim` (when
supported). Not required if test targets are rotational
wkhtmltopdf Generates PDF version of report - download from
http://wkhtmltopdf.org
zip Archives HTML test report into a single zip file
TEST ARTIFACTS
Upon successful completion of testing, the following artifacts will be
produced in the working directory ([test] replaced with one of the test
identifiers - e.g. report-iops.json):
[test].json JSON formatted job results for [test]. Each test provides
different result metrics. This file contains a single
level hash of key/value pairs representing these metrics
fio-[test].json JSON formatted fio job results for [test]. Jobs are
in run order each with a unique job name. For workload
independent preconditioning, the job name uses the
format 'wipc-N', where N is the preconditioning pass
(i.e. N=1 is first pass, N=2 second). Unless
--precondition_passes is otherwise specified, only 2 wipc
JSON files should be present (each representing one of the
2X preconditioning tests). For workload dependent
preconditioning and other testing, the job name is set by
the test. For example, for the IOPS test, the job name
format is 'xN-[rw]-[bs]-rand', where N is iteration
number (1-25+), [rw] is the read/write ratio (separated by
underscore) and [bs] is block size. Jobs that fall within
the steady state measurement window will have the suffix
'-ssmw' (e.g. x5-0_100-4k-rand-ssmw). there may be up to
10 fio-[test].json corresponding with each of the tests
and 2 files for throughput: test-throughput-128k.json and
test-throughput-1024k.json
report.zip HTML test report (open index.html). The report design
and layout is based on the SNIA test specification. In
addition, this archive contains source gnuplot scripts and
data files for report graphs. Graphs are rendered in svg
format
report.pdf PDF version of the test report (wkhtmltopdf used to
generate this version of the report)
collectd-rrd.zip collectd RRD files (see --collectd_rrd)
SAVE SCHEMA
The following columns are included in CSV files/tables generated by save.sh.
Indexed MySQL/PostgreSQL columns are identified by *. Columns without
descriptions are documented as runtime parameters above. Data types and
indexing used is documented in save/schema/*.json. Columns can be removed using
the save.sh --remove parameter
COMMON COLUMNS (included in all CSV/tables):
active_range
benchmark_version: [benchmark version]
collectd_rrd: [URL to zip file containing collectd rrd files]
fio_version: [fio version]
iteration: [iteration number (used with incremental result directories)]
meta_compute_service
meta_compute_service_id*
meta_cpu: [CPU model info]
meta_cpu_cache: [CPU cache]
meta_cpu_cores: [# of CPU cores]
meta_cpu_speed: [CPU clock speed (MHz)]
meta_drive_interface
meta_drive_model
meta_drive_type
meta_instance_id*
meta_hostname: [system under test (SUT) hostname]
meta_memory
meta_memory_gb: [memory in gigabytes]
meta_memory_mb: [memory in megabyets]
meta_os_info: [operating system name and version]
meta_provider
meta_provider_id*
meta_region*
meta_resource_id
meta_run_id
meta_storage_config*
meta_storage_vol_info
meta_test_id*
meta_test_sw
noprecondition
nopurge
norandom
nosecureerase
notrim
nozerofill
oio_per_thread: [queue depth per thread]
precondition_passes
purge_methods: [purge methods used for each device]
results_pdf: [URL to results PDF file (if --store option used)]
results_zip: [URL to results ZIP file (if --store option used)]
skip_blocksize
skip_workload
# Note: ss_* columns are not part of fio or wsat tables
ss_average: [mean value of y axis in steady state window]
ss_largest_data_excursion: [largest data excusion in steady state window]
ss_largest_slope_excursion: [largest slope excusion in steady state window]
ss_linear_fit_formula: [steady state linear fit formula]
ss_max_data_excursion: [max allowed data excusion in steady state window]
ss_max_rounds
ss_max_slope_excursion: [max allowed slope excusion in steady state window]
ss_rounds: [steady state window rounds (e.g. 1 to 5)]
ss_slope: [slope in steady state window]
ss_start: [steady state window start round]
ss_stop: [steady state window end round]
ss_verification
ss_y_intercept: [y intercept in steady state window]
target
target_count: [number of targets]
target_size_gb: [average target size in gigabytes]
target_sizes: [sizes of each target]
test*: [test identifier (e.g. iops)]
test_started*: [when the test started]
test_stopped: [when the test ended]
threads: [number of threads per target]
threads_total: [total number of threads]
timeout
wd_sleep_between
wd_test_duration
FIO CSV/TABLES
fio_command: [complete fio command]
iodepth_level_1: [distribution of IO with IO depth <= 1]
iodepth_level_2: [distribution of IO with IO depth 2]
iodepth_level_4: [distribution of IO with IO depth 4]
iodepth_level_8: [distribution of IO with IO depth 8]
iodepth_level_16: [distribution of IO with IO depth 16]
iodepth_level_32: [distribution of IO with IO depth 32]
iodepth_level_gte64: [distribution of IO with IO depth >= 64]
jobname: [fio job name]
latency_us_2: [distribution of IO with latency <= 2 microseconds]
latency_us_4: [distribution of IO with latency >2 and <=4 microseconds]
latency_us_10: [distribution of IO with latency >4 and <=10 microseconds]
latency_us_20: [distribution of IO with latency >10 and <=20 microseconds]
latency_us_50: [distribution of IO with latency >20 and <=50 microseconds]
latency_us_100: [distribution of IO with latency >50 and <=100 microseconds]
latency_us_250: [distribution of IO with latency >100 and <=250 microseconds]
latency_us_500: [distribution of IO with latency >250 and <=500 microseconds]
latency_us_750: [distribution of IO with latency >500 and <=750 microseconds]
latency_us_1000: [distribution of IO with latency >750 and <=1000 microseconds]
latency_ms_2: [distribution of IO with latency >1000 microseconds and <=2 milliseconds]
latency_ms_4: [distribution of IO with latency >2 and <=4 milliseconds]
latency_ms_10: [distribution of IO with latency >4 and <=10 milliseconds]
latency_ms_20: [distribution of IO with latency >10 and <=20 milliseconds]
latency_ms_50: [distribution of IO with latency >20 and <=50 milliseconds]
latency_ms_100: [distribution of IO with latency >50 and <=100 milliseconds]
latency_ms_250: [distribution of IO with latency >100 and <=250 milliseconds]
latency_ms_500: [distribution of IO with latency >250 and <=500 milliseconds]
latency_ms_750: [distribution of IO with latency >500 and <=750 milliseconds]
latency_ms_1000: [distribution of IO with latency >750 and <=1000 milliseconds]
latency_ms_2000: [distribution of IO with latency >1000 and <2000 milliseconds]
latency_ms_gte2000: [distribution of IO with latency >=2000 milliseconds]
majf: [major page faults]
minf: [minor page faults]
read_io_bytes: [read IO - KB]
read_bw: [read bandwidth - KB/s]
read_iops: [read IOPS]
read_runtime: [read runtime - ms]
read_slat_min: [min read submission latency - μs]
read_slat_max: [max read submission latency - μs]
read_slat_mean: [mean read submission latency - μs]
read_slat_stddev: [read submission latency standard deviation - μs]
read_clat_min: [min read completion latency - μs]
read_clat_max: [max read completion latency - μs]
read_clat_mean: [mean read completion latency - μs]
read_clat_stddev: [read completion standard deviation - μs]
read_clat_percentile_1: [1st percentile read completion latency - μs]
read_clat_percentile_5: [5th percentile read completion latency - μs]
read_clat_percentile_10: [10th percentile read completion latency - μs]
read_clat_percentile_20: [20th percentile read completion latency - μs]
read_clat_percentile_30: [30th percentile read completion latency - μs]
read_clat_percentile_40: [40th percentile read completion latency - μs]
read_clat_percentile_50: [50th percentile read completion latency - μs]
read_clat_percentile_60: [60th percentile read completion latency - μs]
read_clat_percentile_70: [70th percentile read completion latency - μs]
read_clat_percentile_80: [80th percentile read completion latency - μs]
read_clat_percentile_90: [90th percentile read completion latency - μs]
read_clat_percentile_95: [95th percentile read completion latency - μs]
read_clat_percentile_99: [99th percentile read completion latency - μs]
read_clat_percentile_99_5: [99.5th percentile read completion latency - μs]
read_clat_percentile_99_9: [99.9th percentile read completion latency - μs]
read_clat_percentile_99_95: [99.95th percentile read completion latency - μs]
read_clat_percentile_99_99: [99.99th percentile read completion latency - μs]
read_lat_min: [min total read latency - μs]
read_lat_max: [max total read latency - μs]
read_lat_mean: [mean total read latency - μs]
read_lat_stddev: [total read latency standard deviation - μs]
read_bw_min: [min read bandwidth - KB/s]
read_bw_max: [max read bandwidth - KB/s]
read_bw_agg: [aggregate read bandwidth - KB/s]
read_bw_mean: [mean read bandwidth - KB/s]
read_bw_dev: [read bandwidth standard deviation - KB/s]
started: [when the fio job started]
stopped: [when the fio job ended]
usr_cpu: [user CPU usage]
sys_cpu: [system CPU usage]
write_io_bytes: [write IO - KB]
write_bw: [write bandwidth - KB/s]
write_iops: [write IOPS]
write_runtime: [write runtime - ms]
write_slat_min: [min write submission latency - μs]
write_slat_max: [max write submission latency - μs]
write_slat_mean: [mean write submission latency - μs]
write_slat_stddev: [write submission latency standard deviation - μs]
write_clat_min: [min write completion latency - μs]
write_clat_max: [max write completion latency - μs]
write_clat_mean: [mean write completion latency - μs]
write_clat_stddev: [write completion standard deviation - μs]
write_clat_percentile_1: [1st percentile write completion latency - μs]
write_clat_percentile_5: [5th percentile write completion latency - μs]
write_clat_percentile_10: [10th percentile write completion latency - μs]
write_clat_percentile_20: [20th percentile write completion latency - μs]
write_clat_percentile_30: [30th percentile write completion latency - μs]
write_clat_percentile_40: [40th percentile write completion latency - μs]
write_clat_percentile_50: [50th percentile write completion latency - μs]
write_clat_percentile_60: [60th percentile write completion latency - μs]
write_clat_percentile_70: [70th percentile write completion latency - μs]
write_clat_percentile_80: [80th percentile write completion latency - μs]
write_clat_percentile_90: [90th percentile write completion latency - μs]
write_clat_percentile_95: [95th percentile write completion latency - μs]
write_clat_percentile_99: [99th percentile write completion latency - μs]
write_clat_percentile_99_5: [99.5th percentile write completion latency - μs]
write_clat_percentile_99_9: [99.9th percentile write completion latency - μs]
write_clat_percentile_99_95: [99.95th percentile write completion latency - μs]
write_clat_percentile_99_99: [99.99th percentile write completion latency - μs]
write_lat_min: [min total write latency - μs]
write_lat_max: [max total write latency - μs]
write_lat_mean: [mean total write latency - μs]
write_lat_stddev: [total write latency standard deviation - μs]
write_bw_min: [min write bandwidth - KB/s]
write_bw_max: [max write bandwidth - KB/s]
write_bw_agg: [aggregate write bandwidth - KB/s]
write_bw_mean: [mean write bandwidth - KB/s]
write_bw_dev: [write bandwidth standard deviation - KB/s]
HIR CSV/TABLES:
hir_steady_state_iops: [mean 4k rand write IOPS during steady state phase of preconditioning]
hir_wait_5s_iops: [mean 4k rand write IOPS during testing with 5 second wait intervals]
hir_wait_10s_iops: [mean 4k rand write IOPS during testing with 10 second wait intervals]
hir_wait_15s_iops: [mean 4k rand write IOPS during testing with 15 second wait intervals]
hir_wait_25s_iops: [mean 4k rand write IOPS during testing with 25 second wait intervals]
hir_wait_50s_iops: [mean 4k rand write IOPS during testing with 50 second wait intervals]
IOPS CSV/TABLES:
iops_1m_100_0: [mean 1m read IOPS in steady state]
iops_128k_100_0: [mean 128k read IOPS in steady state]
iops_64k_100_0: [mean 64k read IOPS in steady state]
iops_32k_100_0: [mean 32k read IOPS in steady state]
iops_16k_100_0: [mean 16k read IOPS in steady state]
iops_8k_100_0: [mean 8k read IOPS in steady state]
iops_4k_100_0: [mean 4k read IOPS in steady state]
iops_512b_100_0: [mean 512b read IOPS in steady state]
iops_1m_95_5: [mean 1m 95/5 rw IOPS in steady state]
iops_128k_95_5: [mean 128k 95/5 rw IOPS in steady state]
iops_64k_95_5: [mean 64k 95/5 rw IOPS in steady state]
iops_32k_95_5: [mean 32k 95/5 rw IOPS in steady state]
iops_16k_95_5: [mean 16k 95/5 rw IOPS in steady state]
iops_8k_95_5: [mean 8k 95/5 rw IOPS in steady state]
iops_4k_95_5: [mean 4k 95/5 rw IOPS in steady state]
iops_512b_95_5: [mean 512b 95/5 rw IOPS in steady state]
iops_1m_65_35: [mean 1m 65/35 rw IOPS in steady state]
iops_128k_65_35: [mean 128k 65/35 rw IOPS in steady state]
iops_64k_65_35: [mean 64k 65/35 rw IOPS in steady state]
iops_32k_65_35: [mean 32k 65/35 rw IOPS in steady state]
iops_16k_65_35: [mean 16k 65/35 rw IOPS in steady state]
iops_8k_65_35: [mean 8k 65/35 rw IOPS in steady state]
iops_4k_65_35: [mean 4k 65/35 rw IOPS in steady state]
iops_512b_65_35: [mean 512b 65/35 rw IOPS in steady state]
iops_1m_50_50: [mean 1m 50/50 rw IOPS in steady state]
iops_128k_50_50: [mean 128k 50/50 rw IOPS in steady state]
iops_64k_50_50: [mean 64k 50/50 rw IOPS in steady state]
iops_32k_50_50: [mean 32k 50/50 rw IOPS in steady state]
iops_16k_50_50: [mean 16k 50/50 rw IOPS in steady state]
iops_8k_50_50: [mean 8k 50/50 rw IOPS in steady state]
iops_4k_50_50: [mean 4k 50/50 rw IOPS in steady state]
iops_512b_50_50: [mean 512b 50/50 rw IOPS in steady state]
iops_1m_35_65: [mean 1m 35/65 rw IOPS in steady state]
iops_128k_35_65: [mean 128k 35/65 rw IOPS in steady state]
iops_64k_35_65: [mean 64k 35/65 rw IOPS in steady state]
iops_32k_35_65: [mean 32k 35/65 rw IOPS in steady state]
iops_16k_35_65: [mean 16k 35/65 rw IOPS in steady state]
iops_8k_35_65: [mean 8k 35/65 rw IOPS in steady state]
iops_4k_35_65: [mean 4k 35/65 rw IOPS in steady state]
iops_512b_35_65: [mean 512b 35/65 rw IOPS in steady state]
iops_1m_5_95: [mean 1m 5/95 rw IOPS in steady state]
iops_128k_5_95: [mean 128k 5/95 rw IOPS in steady state]
iops_64k_5_95: [mean 64k 5/95 rw IOPS in steady state]
iops_32k_5_95: [mean 32k 5/95 rw IOPS in steady state]
iops_16k_5_95: [mean 16k 5/95 rw IOPS in steady state]
iops_8k_5_95: [mean 8k 5/95 rw IOPS in steady state]
iops_4k_5_95: [mean 4k 5/95 rw IOPS in steady state]
iops_512b_5_95: [mean 512b 5/95 rw IOPS in steady state]
iops_1m_0_100: [mean 1m write IOPS in steady state]
iops_128k_0_100: [mean 128k write IOPS in steady state]
iops_64k_0_100: [mean 64k write IOPS in steady state]
iops_32k_0_100: [mean 32k write IOPS in steady state]
iops_16k_0_100: [mean 16k write IOPS in steady state]
iops_8k_0_100: [mean 8k write IOPS in steady state]
iops_4k_0_100: [mean 4k write IOPS in steady state]
iops_512b_0_100: [mean 512b write IOPS in steady state]
LATENCY CSV/TABLES:
latency_8k_100_0_mean: [mean 8k read latency]
latency_8k_100_0_max: [max 8k read latency]
latency_4k_100_0_mean: [mean 4k read latency]
latency_4k_100_0_max: [max 8k read latency]
latency_512b_100_0_mean: [mean 512b read latency]
latency_512b_100_0_max: [max 512b read latency]
latency_8k_65_35_mean: [mean 8k 65/35 rw latency]
latency_8k_65_35_max: [max 8k 65/35 rw latency]
latency_4k_65_35_mean: [mean 4k 65/35 rw latency]
latency_4k_65_35_max: [max 4k 65/35 rw latency]
latency_512b_65_35_mean: [mean 512b 65/35 rw latency]
latency_512b_65_35_max: [max 512b 65/35 rw latency]
latency_8k_0_100_mean: [mean 8k write latency]
latency_8k_0_100_max: [max 8k write latency]
latency_4k_0_100_mean: [mean 4k write latency]
latency_4k_0_100_max: [max 4k write latency]
latency_512b_0_100_mean: [mean 512b write latency]
latency_512b_0_100_max: [max 512b write latency]
THROUGHPUT CSV/TABLES:
throughput_1024k_100_0: [mean 1024k read throughput - MB/s]
throughput_1024k_0_100: [mean 1024k write throughput - MB/s]
throughput_128k_100_0: [mean 128k write throughput - MB/s]
throughput_128k_0_100: [mean 128k write throughput - MB/s]
WSAT CSV/TABLES
wsat_iops: [mean 4k write IOPS in steady state]
USAGE
# perform IOPS test against device /dev/sdc
./run.sh --target=/dev/sdc --test=iops
# perform IOPS, Latency and Throughput tests against /dev/sdc and /dev/sdd
# concurrently using a maximum of [num CPU cores]*2 and 32 OIO per thread
./run.sh --target=/dev/sdc --target=/dev/sdd --test=iops --test=latency --test=throughput --threads="{cpu}*2" --oio_per_thread=32
# perform IOPS test against device /dev/sdc but skip the purge step
./run.sh --target=/dev/sdc --test=iops --nopurge
# perform IOPS test against device /dev/sdc but skip the purge and workload
# independent preconditioning
./run.sh --target=/dev/sdc --test=iops --nopurge --noprecondition
# perform 5 iterations of the same IOPS test above
for i in {1..5}; do mkdir -p ~/block-storage-testing/$i; ./run.sh --target=/dev/sdc --test=iops --nopurge --noprecondition --output ~/block-storage-testing/$i; done
# save.sh saves results to CSV, MySQL, PostgreSQL, BigQuery or via HTTP
# callback. It can also save artifacts (PDF and ZIP reports) to S3, Azure
# Blob Storage or Google Cloud Storage
# save results to CSV files
./save.sh
# save results from 5 iterations text example above
./save.sh ~/block-storage-testing
# save results to a PostgreSQL database
./save --db postgresql --db_user dbuser --db_pswd dbpass --db_host db.mydomain.com --db_name benchmarks
# save results to BigQuery and artifacts (PDF and ZIP reports) to S3
./save --db bigquery --db_name benchmark_dataset --store s3 --store_key THISIH5TPISAEZIJFAKE --store_secret thisNoat1VCITCGggisOaJl3pxKmGu2HMKxxfake --store_container benchmarks1234