initial benchmarks #5
Conversation
|
Hello @emmanuelle! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:
Comment last updated at 2020-08-09 19:46:41 UTC |
| return times, non_tested_functions | ||
|
|
||
|
|
||
| # ----------------------- Run benchmark ------------------------- |
There was a problem hiding this comment.
Right :-). I'm also hesitant whether to merge together the two scripts.
There was a problem hiding this comment.
Might as well, for now, and keep it self contained. Neither is huge.
| img2 = data.binary_blobs(length=l1, volume_fraction=0.3, seed=10).astype(np.uint8) | ||
|
|
||
|
|
||
| parameters = { |
There was a problem hiding this comment.
I like this. It's a pretty nice, compact way to do this.
There was a problem hiding this comment.
Cool. I'm indeed looking for a middle ground where we can test as many functions as possible but without writing too much code for this.
benchmarks/benchmark_suite.py
Outdated
| print("-" * 70) | ||
| for func in inspect.getmembers(submodule, inspect.isfunction): | ||
| if func[0] in times: | ||
| print(func[0], times[func[0]]) |
There was a problem hiding this comment.
The main thing would be to display more clearly. Maybe you could make a plotly scatterplot with a log axis and function names on hover? =) And maybe coloured by module. =)
|
@emmanuelle as usual I love this! Main thing would be to add some dataviz on top. ;) |
|
I updated the script with some visualization, see You need to run the script to get an interactive version :-). I was thinking that maybe it could be possible to turn this script into a gallery example (starting from a smaller image, for example l1=400), with some explanations about what controls the execution time (most of our functions scale as O(N) with the number of pixels, but the prefactor varies wildly since there can be a large number of iterations, costly operations like exp for each pixel, etc). How would that be? |
|
I improved the script so that the result of the line profiler is displayed in the hover. The idea is that it can help getting a quick idea of where the time is spent. The result can be seen on http://sd-118541.dedibox.fr/benchmark_profile_skimage.html |
This PR shares a script I have been working on to get a feeling of which scikit-image functions take more time than others (and are therefore worth optimizing).
The approach is "quick and dirty" (for very accurate benchmarks we have asv), the idea being to compare all functions for the same image (or at least an image of the same size). The approach has several shortcomings (for some functions the test image might be really silly so that the measured time is not representative of a normal usecase), but it's a first attempt.
I have several ideas about how to improve this but I'm first interested in feedback @grlee77 @jni
Below is the current output of the script
Functions which could not be tested
corner_orientations
draw_multiblock_lbp
greycomatrix
greycoprops
haar_like_feature
haar_like_feature_coord
local_binary_pattern
masked_register_translation
match_descriptors
match_template
multiblock_lbp
peak_local_max
plot_matches
register_translation
structure_tensor_eigvals
gabor_kernel
inverse
wiener
approximate_polygon
grid_points_in_poly
marching_cubes
marching_cubes_classic
marching_cubes_lewiner
mesh_surface_area
moments_coords
moments_coords_central
moments_normalized
perimeter
points_in_poly
profile_line
subdivide_polygon
adapted_rand_error
reconstruction
skeletonize
watershed
richardson_lucy
unsupervised_wiener
wiener
active_contour
checkerboard_level_set
circle_level_set
disk_level_set
mark_boundaries
morphological_chan_vese
morphological_geodesic_active_contour
random_walker
ifrt2
iradon
resize
warp
warp_coords
crop
map_array
montage
pad
Sorted by increasing execution time
pyramid_laplacian 2.054000105999876e-06
_guess_spatial_dimensions 2.347999952689861e-06
pyramid_gaussian 4.665999995268066e-06
order_angles_golden_ratio 5.5100001645769225e-06
dtype_limits 7.336999942708644e-06
img_as_ubyte 6.257900008677098e-05
structure_tensor_eigenvalues 0.0001295199999731267
invert 0.00026143700006286963
img_as_uint 0.0007241559999329183
threshold_mean 0.0007569779999698767
img_as_bool 0.000854823000054239
moments_hu 0.000860465999949156
img_as_float32 0.0010989429999881395
flood 0.0013119760000108727
hessian_matrix_eigvals 0.0018025609999767767
img_as_float64 0.0019991529998151236
flood_fill 0.00210337700002583
centroid 0.0025426130000028024
mean_squared_error 0.0026007320000189793
unique_rows 0.0030487700000776385
threshold_isodata 0.004387242999996488
img_as_float 0.004448335000006409
img_as_int 0.0052689359999931185
threshold_triangle 0.005769133999990572
histogram 0.006056851000039387
normalized_root_mse 0.006342520999965018
moments 0.006695659000001797
inertia_tensor 0.007631108999987646
threshold_multiotsu 0.007699045999970622
moments_central 0.007767607999994652
threshold_otsu 0.008106948999966335
peak_signal_noise_ratio 0.008312294999996084
binary_dilation 0.008779259999982969
block_reduce 0.008840405000000828
is_low_contrast 0.008971159999987321
threshold_yen 0.009236493000003065
compare_images 0.010282098000061524
regionprops 0.010392365999962294
roberts_pos_diag 0.010650161000000935
roberts_neg_diag 0.011252759999990758
inertia_tensor_eigvals 0.011689529999955539
adjust_gamma 0.011953222999977697
regionprops_table 0.012241316000029201
cumulative_distribution 0.013172133000011854
remove_small_objects 0.013718206000021382
erosion 0.013957274999995661
downscale_local_mean 0.014171291000025121
label 0.014663558999984616
scharr_v 0.014881955000021208
sobel_v 0.015023283999994419
binary_erosion 0.01506791299999577
dilation 0.015579463000051419
scharr_h 0.015975135999951817
rescale_intensity 0.016481907000013507
sobel_h 0.016553729999998268
adjust_log 0.01736698500002376
integral_image 0.019766213000025346
closing 0.02113654999999426
binary_opening 0.022080103999996936
black_tophat 0.022521785999970234
prewitt_v 0.022849717999974928
laplace 0.023063979000028212
white_tophat 0.023776355999984844
prewitt_h 0.023856552999973246
find_boundaries 0.02452055300000211
local_minima 0.024819010999976854
remove_small_holes 0.026253840000038053
farid_v 0.027667088999976386
gaussian 0.028063400000007732
local_maxima 0.02829098699999122
binary_closing 0.028351206999957412
farid_h 0.02839656699995885
roberts 0.02929483500003016
pyramid_reduce 0.03072875199995906
adjust_sigmoid 0.03152125100001513
unsharp_mask 0.031763064999950075
contingency_table 0.03421420600000147
scharr 0.03423365499997999
rotate 0.034322747000032905
sobel 0.03501391600002535
opening 0.03633150199999591
apply_hysteresis_threshold 0.03680344399998603
shannon_entropy 0.038076080999985606
estimate_sigma 0.04107598799998868
hessian_matrix 0.04514324399997349
variation_of_information 0.04567708700000139
prewitt 0.04701683600001161
rescale 0.04860022200000458
corner_moravec 0.05078498899996475
warp_polar 0.051725067000006675
hessian_matrix_det 0.054374754999969355
relabel_sequential 0.054947781000009854
difference_of_gaussians 0.05529835300001196
random_noise 0.06253589600009946
corner_kitchen_rosenfeld 0.06564494399998466
farid 0.07190258199995014
convex_hull_image 0.07421858800000791
threshold_minimum 0.08598453400003336
corner_harris 0.08689929399997709
equalize_hist 0.08695479099998238
threshold_niblack 0.09053989799997453
corner_foerstner 0.09599825399999418
threshold_sauvola 0.10285040299999082
denoise_wavelet 0.10700316199995541
rank_order 0.12171512800000528
find_contours 0.12699810300000536
corner_shi_tomasi 0.12879965799999127
match_histograms 0.13347542600001816
structure_tensor 0.1346472780000454
inverse_gaussian_gradient 0.13671372199996767
expand_labels 0.13787843400007205
join_segmentations 0.14492319800001496
median 0.15073970299999928
clear_border 0.15465642699996351
structural_similarity 0.16366587300001356
phase_cross_correlation 0.16665595400002076
threshold_li 0.17133916000000227
equalize_adapthist 0.17480362499998137
threshold_local 0.19413144799995052
shape_index 0.20180447600000662
corner_fast 0.21287866400001576
swirl 0.23490168600005745
pyramid_expand 0.23864449100005913
hough_circle 0.24992335300009927
gabor 0.25993827899998223
denoise_tv_bregman 0.3135844419999785
canny 0.39808771399998477
probabilistic_hough_line 0.6443802329999926
skeletonize_3d 0.6868357120000042
hog 0.7362922610000169
slic 0.8500977339999736
unwrap_phase 0.992899833000024
sato 0.9942164020000064
denoise_bilateral 1.1008444149999832
hough_line 1.284792312000036
try_all_threshold 1.2923166190000188
cycle_spin 1.349445951000007
meijering 1.3647139369999763
frangi 1.4968320290000179
denoise_tv_chambolle 1.5301364949999652
h_minima 1.553363030000014
hessian 1.5598298699999873
h_maxima 1.569264443999998
blob_doh 2.0725020419999964
blob_dog 2.225544811999953
denoise_nl_means 2.2459059309999247
convex_hull_object 2.2874292649999575
medial_axis 2.42722605299997
watershed 2.4539618760001076
hausdorff_distance 2.8771912820000125
daisy 3.691399289000003
felzenszwalb 3.7035217809999494
optical_flow_tvl1 4.081598399999962
corner_peaks 4.839632607999988
blob_log 5.5202517639999655
radon 5.685521913000002
thin 12.606799143000046
frt2 18.860716036999975
chan_vese 19.578543106999973
area_closing 19.970794318000003
max_tree 22.986526878999996
area_opening 23.382182682999996
max_tree_local_maxima 24.26655013300001
quickshift 25.16158812499998
diameter_closing 34.45153475999996
diameter_opening 35.154202027
iradon_sart 115.38031583299994
inpaint_biharmonic 362.703145866
Sorted by subpackage
skimage.exposure
adjust_gamma 0.011953222999977697
adjust_log 0.01736698500002376
adjust_sigmoid 0.03152125100001513
cumulative_distribution 0.013172133000011854
equalize_adapthist 0.17480362499998137
equalize_hist 0.08695479099998238
histogram 0.006056851000039387
is_low_contrast 0.008971159999987321
match_histograms 0.13347542600001816
rescale_intensity 0.016481907000013507
skimage.feature
blob_dog 2.225544811999953
blob_doh 2.0725020419999964
blob_log 5.5202517639999655
canny 0.39808771399998477
corner_fast 0.21287866400001576
corner_foerstner 0.09599825399999418
corner_harris 0.08689929399997709
corner_kitchen_rosenfeld 0.06564494399998466
corner_moravec 0.05078498899996475
corner_peaks 4.839632607999988
corner_shi_tomasi 0.12879965799999127
daisy 3.691399289000003
hessian_matrix 0.04514324399997349
hessian_matrix_det 0.054374754999969355
hessian_matrix_eigvals 0.0018025609999767767
hog 0.7362922610000169
shape_index 0.20180447600000662
structure_tensor 0.1346472780000454
structure_tensor_eigenvalues 0.0001295199999731267
skimage.filters
_guess_spatial_dimensions 2.347999952689861e-06
apply_hysteresis_threshold 0.03680344399998603
difference_of_gaussians 0.05529835300001196
farid 0.07190258199995014
farid_h 0.02839656699995885
farid_v 0.027667088999976386
frangi 1.4968320290000179
gabor 0.25993827899998223
gaussian 0.028063400000007732
hessian 1.5598298699999873
laplace 0.023063979000028212
median 0.15073970299999928
meijering 1.3647139369999763
prewitt 0.04701683600001161
prewitt_h 0.023856552999973246
prewitt_v 0.022849717999974928
rank_order 0.12171512800000528
roberts 0.02929483500003016
roberts_neg_diag 0.011252759999990758
roberts_pos_diag 0.010650161000000935
sato 0.9942164020000064
scharr 0.03423365499997999
scharr_h 0.015975135999951817
scharr_v 0.014881955000021208
sobel 0.03501391600002535
sobel_h 0.016553729999998268
sobel_v 0.015023283999994419
threshold_isodata 0.004387242999996488
threshold_li 0.17133916000000227
threshold_local 0.19413144799995052
threshold_mean 0.0007569779999698767
threshold_minimum 0.08598453400003336
threshold_multiotsu 0.007699045999970622
threshold_niblack 0.09053989799997453
threshold_otsu 0.008106948999966335
threshold_sauvola 0.10285040299999082
threshold_triangle 0.005769133999990572
threshold_yen 0.009236493000003065
try_all_threshold 1.2923166190000188
unsharp_mask 0.031763064999950075
skimage.measure
block_reduce 0.008840405000000828
centroid 0.0025426130000028024
find_contours 0.12699810300000536
inertia_tensor 0.007631108999987646
inertia_tensor_eigvals 0.011689529999955539
label 0.014663558999984616
moments 0.006695659000001797
moments_central 0.007767607999994652
moments_hu 0.000860465999949156
regionprops 0.010392365999962294
regionprops_table 0.012241316000029201
shannon_entropy 0.038076080999985606
skimage.metrics
contingency_table 0.03421420600000147
hausdorff_distance 2.8771912820000125
mean_squared_error 0.0026007320000189793
normalized_root_mse 0.006342520999965018
peak_signal_noise_ratio 0.008312294999996084
structural_similarity 0.16366587300001356
variation_of_information 0.04567708700000139
skimage.morphology
area_closing 19.970794318000003
area_opening 23.382182682999996
binary_closing 0.028351206999957412
binary_dilation 0.008779259999982969
binary_erosion 0.01506791299999577
binary_opening 0.022080103999996936
black_tophat 0.022521785999970234
closing 0.02113654999999426
convex_hull_image 0.07421858800000791
convex_hull_object 2.2874292649999575
diameter_closing 34.45153475999996
diameter_opening 35.154202027
dilation 0.015579463000051419
erosion 0.013957274999995661
flood 0.0013119760000108727
flood_fill 0.00210337700002583
h_maxima 1.569264443999998
h_minima 1.553363030000014
label 0.014663558999984616
local_maxima 0.02829098699999122
local_minima 0.024819010999976854
max_tree 22.986526878999996
max_tree_local_maxima 24.26655013300001
medial_axis 2.42722605299997
opening 0.03633150199999591
remove_small_holes 0.026253840000038053
remove_small_objects 0.013718206000021382
skeletonize_3d 0.6868357120000042
thin 12.606799143000046
watershed 2.4539618760001076
white_tophat 0.023776355999984844
skimage.registration
optical_flow_tvl1 4.081598399999962
phase_cross_correlation 0.16665595400002076
skimage.restoration
cycle_spin 1.349445951000007
denoise_bilateral 1.1008444149999832
denoise_nl_means 2.2459059309999247
denoise_tv_bregman 0.3135844419999785
denoise_tv_chambolle 1.5301364949999652
denoise_wavelet 0.10700316199995541
estimate_sigma 0.04107598799998868
inpaint_biharmonic 362.703145866
unwrap_phase 0.992899833000024
skimage.segmentation
chan_vese 19.578543106999973
clear_border 0.15465642699996351
expand_labels 0.13787843400007205
felzenszwalb 3.7035217809999494
find_boundaries 0.02452055300000211
flood 0.0013119760000108727
flood_fill 0.00210337700002583
inverse_gaussian_gradient 0.13671372199996767
join_segmentations 0.14492319800001496
quickshift 25.16158812499998
relabel_sequential 0.054947781000009854
slic 0.8500977339999736
watershed 2.4539618760001076
skimage.transform
downscale_local_mean 0.014171291000025121
frt2 18.860716036999975
hough_circle 0.24992335300009927
hough_line 1.284792312000036
integral_image 0.019766213000025346
iradon_sart 115.38031583299994
order_angles_golden_ratio 5.5100001645769225e-06
probabilistic_hough_line 0.6443802329999926
pyramid_expand 0.23864449100005913
pyramid_gaussian 4.665999995268066e-06
pyramid_laplacian 2.054000105999876e-06
pyramid_reduce 0.03072875199995906
radon 5.685521913000002
rescale 0.04860022200000458
rotate 0.034322747000032905
swirl 0.23490168600005745
warp_polar 0.051725067000006675
skimage.util
compare_images 0.010282098000061524
dtype_limits 7.336999942708644e-06
img_as_bool 0.000854823000054239
img_as_float 0.004448335000006409
img_as_float32 0.0010989429999881395
img_as_float64 0.0019991529998151236
img_as_int 0.0052689359999931185
img_as_ubyte 6.257900008677098e-05
img_as_uint 0.0007241559999329183
invert 0.00026143700006286963
random_noise 0.06253589600009946
unique_rows 0.0030487700000776385