-
Notifications
You must be signed in to change notification settings - Fork 0
/
power_perf_tp4.txt
8309 lines (7803 loc) · 254 KB
/
power_perf_tp4.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
GPU clocks set to "(gpuClkMin 800, gpuClkMax 800)" for GPU 00000001:00:00.0
Warning: persistence mode is disabled on device 00000001:00:00.0. See the Known Issues section of the nvidia-smi(1) man page for more information. Run with [--help | -h] switch to get more information on how to enable persistence mode.
GPU clocks set to "(gpuClkMin 800, gpuClkMax 800)" for GPU 00000002:00:00.0
Warning: persistence mode is disabled on device 00000002:00:00.0. See the Known Issues section of the nvidia-smi(1) man page for more information. Run with [--help | -h] switch to get more information on how to enable persistence mode.
GPU clocks set to "(gpuClkMin 800, gpuClkMax 800)" for GPU 00000003:00:00.0
Warning: persistence mode is disabled on device 00000003:00:00.0. See the Known Issues section of the nvidia-smi(1) man page for more information. Run with [--help | -h] switch to get more information on how to enable persistence mode.
GPU clocks set to "(gpuClkMin 800, gpuClkMax 800)" for GPU 00000008:00:00.0
Warning: persistence mode is disabled on device 00000008:00:00.0. See the Known Issues section of the nvidia-smi(1) man page for more information. Run with [--help | -h] switch to get more information on how to enable persistence mode.
GPU clocks set to "(gpuClkMin 800, gpuClkMax 800)" for GPU 00000009:00:00.0
Warning: persistence mode is disabled on device 00000009:00:00.0. See the Known Issues section of the nvidia-smi(1) man page for more information. Run with [--help | -h] switch to get more information on how to enable persistence mode.
GPU clocks set to "(gpuClkMin 800, gpuClkMax 800)" for GPU 0000000A:00:00.0
Warning: persistence mode is disabled on device 0000000A:00:00.0. See the Known Issues section of the nvidia-smi(1) man page for more information. Run with [--help | -h] switch to get more information on how to enable persistence mode.
GPU clocks set to "(gpuClkMin 800, gpuClkMax 800)" for GPU 0000000B:00:00.0
Warning: persistence mode is disabled on device 0000000B:00:00.0. See the Known Issues section of the nvidia-smi(1) man page for more information. Run with [--help | -h] switch to get more information on how to enable persistence mode.
GPU clocks set to "(gpuClkMin 800, gpuClkMax 800)" for GPU 0000000C:00:00.0
Warning: persistence mode is disabled on device 0000000C:00:00.0. See the Known Issues section of the nvidia-smi(1) man page for more information. Run with [--help | -h] switch to get more information on how to enable persistence mode.
All done.
/home/azureuser/miniconda3/envs/jovan_vllm/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
warnings.warn(
GPU clocks set to "(gpuClkMin 1000, gpuClkMax 1000)" for GPU 00000001:00:00.0
Warning: persistence mode is disabled on device 00000001:00:00.0. See the Known Issues section of the nvidia-smi(1) man page for more information. Run with [--help | -h] switch to get more information on how to enable persistence mode.
GPU clocks set to "(gpuClkMin 1000, gpuClkMax 1000)" for GPU 00000002:00:00.0
Warning: persistence mode is disabled on device 00000002:00:00.0. See the Known Issues section of the nvidia-smi(1) man page for more information. Run with [--help | -h] switch to get more information on how to enable persistence mode.
GPU clocks set to "(gpuClkMin 1000, gpuClkMax 1000)" for GPU 00000003:00:00.0
Warning: persistence mode is disabled on device 00000003:00:00.0. See the Known Issues section of the nvidia-smi(1) man page for more information. Run with [--help | -h] switch to get more information on how to enable persistence mode.
GPU clocks set to "(gpuClkMin 1000, gpuClkMax 1000)" for GPU 00000008:00:00.0
Warning: persistence mode is disabled on device 00000008:00:00.0. See the Known Issues section of the nvidia-smi(1) man page for more information. Run with [--help | -h] switch to get more information on how to enable persistence mode.
GPU clocks set to "(gpuClkMin 1000, gpuClkMax 1000)" for GPU 00000009:00:00.0
Warning: persistence mode is disabled on device 00000009:00:00.0. See the Known Issues section of the nvidia-smi(1) man page for more information. Run with [--help | -h] switch to get more information on how to enable persistence mode.
GPU clocks set to "(gpuClkMin 1000, gpuClkMax 1000)" for GPU 0000000A:00:00.0
Warning: persistence mode is disabled on device 0000000A:00:00.0. See the Known Issues section of the nvidia-smi(1) man page for more information. Run with [--help | -h] switch to get more information on how to enable persistence mode.
GPU clocks set to "(gpuClkMin 1000, gpuClkMax 1000)" for GPU 0000000B:00:00.0
Warning: persistence mode is disabled on device 0000000B:00:00.0. See the Known Issues section of the nvidia-smi(1) man page for more information. Run with [--help | -h] switch to get more information on how to enable persistence mode.
GPU clocks set to "(gpuClkMin 1000, gpuClkMax 1000)" for GPU 0000000C:00:00.0
Warning: persistence mode is disabled on device 0000000C:00:00.0. See the Known Issues section of the nvidia-smi(1) man page for more information. Run with [--help | -h] switch to get more information on how to enable persistence mode.
All done.
GPU clocks set to "(gpuClkMin 1200, gpuClkMax 1200)" for GPU 00000001:00:00.0
Warning: persistence mode is disabled on device 00000001:00:00.0. See the Known Issues section of the nvidia-smi(1) man page for more information. Run with [--help | -h] switch to get more information on how to enable persistence mode.
GPU clocks set to "(gpuClkMin 1200, gpuClkMax 1200)" for GPU 00000002:00:00.0
Warning: persistence mode is disabled on device 00000002:00:00.0. See the Known Issues section of the nvidia-smi(1) man page for more information. Run with [--help | -h] switch to get more information on how to enable persistence mode.
GPU clocks set to "(gpuClkMin 1200, gpuClkMax 1200)" for GPU 00000003:00:00.0
Warning: persistence mode is disabled on device 00000003:00:00.0. See the Known Issues section of the nvidia-smi(1) man page for more information. Run with [--help | -h] switch to get more information on how to enable persistence mode.
GPU clocks set to "(gpuClkMin 1200, gpuClkMax 1200)" for GPU 00000008:00:00.0
Warning: persistence mode is disabled on device 00000008:00:00.0. See the Known Issues section of the nvidia-smi(1) man page for more information. Run with [--help | -h] switch to get more information on how to enable persistence mode.
GPU clocks set to "(gpuClkMin 1200, gpuClkMax 1200)" for GPU 00000009:00:00.0
Warning: persistence mode is disabled on device 00000009:00:00.0. See the Known Issues section of the nvidia-smi(1) man page for more information. Run with [--help | -h] switch to get more information on how to enable persistence mode.
GPU clocks set to "(gpuClkMin 1200, gpuClkMax 1200)" for GPU 0000000A:00:00.0
Warning: persistence mode is disabled on device 0000000A:00:00.0. See the Known Issues section of the nvidia-smi(1) man page for more information. Run with [--help | -h] switch to get more information on how to enable persistence mode.
GPU clocks set to "(gpuClkMin 1200, gpuClkMax 1200)" for GPU 0000000B:00:00.0
Warning: persistence mode is disabled on device 0000000B:00:00.0. See the Known Issues section of the nvidia-smi(1) man page for more information. Run with [--help | -h] switch to get more information on how to enable persistence mode.
GPU clocks set to "(gpuClkMin 1200, gpuClkMax 1200)" for GPU 0000000C:00:00.0
Warning: persistence mode is disabled on device 0000000C:00:00.0. See the Known Issues section of the nvidia-smi(1) man page for more information. Run with [--help | -h] switch to get more information on how to enable persistence mode.
All done.
Open ttft file
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.11319869756698608
P50 ttft = 0.11114645004272461
P99 ttft = 0.17014038085937502
Average tbt = 0.03771794637044271
P50 tbt = 0.03777334690093994
P99 tbt = 0.03784971809387207
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.12600683030628024
P50 ttft = 0.11794281005859375
P99 ttft = 0.2181403160095215
Average tbt = 0.0390210197085426
P50 tbt = 0.039167165756225586
P99 tbt = 0.03938567161560059
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.12648710523332868
P50 ttft = 0.11470746994018555
P99 ttft = 0.21163368225097656
Average tbt = 0.04004070690699986
P50 tbt = 0.0400294303894043
P99 tbt = 0.04075189971923828
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.12678403970671864
P50 ttft = 0.1171870231628418
P99 ttft = 0.2112635612487793
Average tbt = 0.04042845935356327
P50 tbt = 0.04057464599609375
P99 tbt = 0.04091789245605469
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.1372701358795166
P50 ttft = 0.1215202808380127
P99 ttft = 0.22727046966552727
Average tbt = 0.04904520034790043
P50 tbt = 0.0411038875579834
P99 tbt = 0.12981291723251343
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.14092546701431274
P50 ttft = 0.12427616119384766
P99 ttft = 0.2387251257896423
Average tbt = 0.08690766319632555
P50 tbt = 0.09557235240936353
P99 tbt = 0.13291303777694702
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.14663311553328004
P50 ttft = 0.13139057159423828
P99 ttft = 0.2554704856872559
Average tbt = 0.10084656362664217
P50 tbt = 0.13164911270141602
P99 tbt = 0.13375374603271484
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.15308151762169528
P50 ttft = 0.13227295875549316
P99 ttft = 0.2882099628448482
Average tbt = 0.1140109584992191
P50 tbt = 0.13434896469116211
P99 tbt = 0.1350197629928589
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.15971228416929853
P50 ttft = 0.13157153129577637
P99 ttft = 0.3073701596260069
Average tbt = 0.12047737456382593
P50 tbt = 0.14072883129119873
P99 tbt = 0.14267777490615846
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.16352070172627767
P50 ttft = 0.1424410343170166
P99 ttft = 0.30027615547180175
Average tbt = 0.14170959881373826
P50 tbt = 0.14320645332336426
P99 tbt = 0.23016639328002936
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.17587432817176538
P50 ttft = 0.15436196327209473
P99 ttft = 0.31752204656600946
Average tbt = 0.17050420840581282
P50 tbt = 0.1533505916595459
P99 tbt = 0.23556254148483277
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.17928851352018468
P50 ttft = 0.16330361366271973
P99 ttft = 0.36242682456970204
Average tbt = 0.18169099623415655
P50 tbt = 0.19186806678772023
P99 tbt = 0.2903754148483282
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.2104359093834372
P50 ttft = 0.2062239646911621
P99 ttft = 0.4049320220947268
Average tbt = 0.18718645467477701
P50 tbt = 0.20460655689239549
P99 tbt = 0.33863389968872076
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.2213423148445461
P50 ttft = 0.217931866645813
P99 ttft = 0.4140385127067565
Average tbt = 0.2157544671625334
P50 tbt = 0.19244735240936317
P99 tbt = 0.43065946626663315
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.0878857175509135
P50 ttft = 0.07775306701660156
P99 ttft = 0.14176300048828128
Average tbt = 0.0305445392926534
P50 tbt = 0.030547261238098145
P99 tbt = 0.030727420330047606
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.09586084456670851
P50 ttft = 0.09174275398254395
P99 ttft = 0.15089664459228516
Average tbt = 0.03143358911786761
P50 tbt = 0.03159451484680176
P99 tbt = 0.03177043914794922
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.10654382705688477
P50 ttft = 0.09845829010009766
P99 ttft = 0.16304396152496337
Average tbt = 0.03236257689339775
P50 tbt = 0.03211245536804199
P99 tbt = 0.033237707138061526
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.10574323956559344
P50 ttft = 0.09707975387573242
P99 ttft = 0.168662166595459
Average tbt = 0.03279660969245724
P50 tbt = 0.03310627937316894
P99 tbt = 0.0335634994506836
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.11101660251617432
P50 ttft = 0.1038891077041626
P99 ttft = 0.18038872003555295
Average tbt = 0.033348922729492185
P50 tbt = 0.03332316875457764
P99 tbt = 0.03380179357528686
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.11188296601176262
P50 ttft = 0.10249900817871094
P99 ttft = 0.18390941858291618
Average tbt = 0.040715518593788164
P50 tbt = 0.03372371196746826
P99 tbt = 0.1068762378692627
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.1144978934771394
P50 ttft = 0.10243964195251465
P99 ttft = 0.19513699531555176
Average tbt = 0.05851842605904367
P50 tbt = 0.03589377403259277
P99 tbt = 0.1085417652130127
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.11712066236748753
P50 ttft = 0.10616421699523926
P99 ttft = 0.18957434654235827
Average tbt = 0.07989221653306347
P50 tbt = 0.10781855583190919
P99 tbt = 0.11073372268676758
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.12273730115687594
P50 ttft = 0.10545659065246582
P99 ttft = 0.213087203502655
Average tbt = 0.09401071122352114
P50 tbt = 0.11022884845733644
P99 tbt = 0.11182452535629273
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.12446425982883998
P50 ttft = 0.11065173149108887
P99 ttft = 0.21252430915832518
Average tbt = 0.10000298999604726
P50 tbt = 0.11183180809020997
P99 tbt = 0.11519450187683108
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.12678288751178318
P50 ttft = 0.1081538200378418
P99 ttft = 0.23864935159683226
Average tbt = 0.1113122445565683
P50 tbt = 0.11724183559417725
P99 tbt = 0.15804887771606502
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.13902931453801004
P50 ttft = 0.11728620529174805
P99 ttft = 0.2539244842529297
Average tbt = 0.11561554980879067
P50 tbt = 0.1185689926147461
P99 tbt = 0.19347869873046875
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.15194795061560237
P50 ttft = 0.1416710615158081
P99 ttft = 0.3472672224044801
Average tbt = 0.1225730924045339
P50 tbt = 0.12477188110351567
P99 tbt = 0.20087045907974246
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.1629720459813657
P50 ttft = 0.14848363399505615
P99 ttft = 0.31643934488296505
Average tbt = 0.1354531516199528
P50 tbt = 0.1325216770172119
P99 tbt = 0.19926854848861703
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.07785618305206299
P50 ttft = 0.0690920352935791
P99 ttft = 0.1235714387893677
Average tbt = 0.026190396149953207
P50 tbt = 0.026210379600524903
P99 tbt = 0.026342435359954834
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.08350412050882976
P50 ttft = 0.07956767082214355
P99 ttft = 0.13593587875366211
Average tbt = 0.026626591455368772
P50 tbt = 0.026512289047241212
P99 tbt = 0.027123260498046874
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.0881734916142055
P50 ttft = 0.08157229423522949
P99 ttft = 0.14109213829040526
Average tbt = 0.027686426980154857
P50 tbt = 0.02755885124206543
P99 tbt = 0.028475714683532716
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.09139869271255122
P50 ttft = 0.08422064781188965
P99 ttft = 0.13826675415039064
Average tbt = 0.02797388216344322
P50 tbt = 0.027772855758666993
P99 tbt = 0.029051189422607423
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.09094487190246582
P50 ttft = 0.08373892307281494
P99 ttft = 0.13910701990127564
Average tbt = 0.028741402626037603
P50 tbt = 0.028736472129821777
P99 tbt = 0.029540473937988285
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.0943126268684864
P50 ttft = 0.0873647928237915
P99 ttft = 0.14548654794692992
Average tbt = 0.031011937558651004
P50 tbt = 0.02918059825897217
P99 tbt = 0.06762960100174004
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.09210110037294153
GPU clocks set to "(gpuClkMin 1400, gpuClkMax 1400)" for GPU 00000001:00:00.0
Warning: persistence mode is disabled on device 00000001:00:00.0. See the Known Issues section of the nvidia-smi(1) man page for more information. Run with [--help | -h] switch to get more information on how to enable persistence mode.
GPU clocks set to "(gpuClkMin 1400, gpuClkMax 1400)" for GPU 00000002:00:00.0
Warning: persistence mode is disabled on device 00000002:00:00.0. See the Known Issues section of the nvidia-smi(1) man page for more information. Run with [--help | -h] switch to get more information on how to enable persistence mode.
GPU clocks set to "(gpuClkMin 1400, gpuClkMax 1400)" for GPU 00000003:00:00.0
Warning: persistence mode is disabled on device 00000003:00:00.0. See the Known Issues section of the nvidia-smi(1) man page for more information. Run with [--help | -h] switch to get more information on how to enable persistence mode.
GPU clocks set to "(gpuClkMin 1400, gpuClkMax 1400)" for GPU 00000008:00:00.0
Warning: persistence mode is disabled on device 00000008:00:00.0. See the Known Issues section of the nvidia-smi(1) man page for more information. Run with [--help | -h] switch to get more information on how to enable persistence mode.
GPU clocks set to "(gpuClkMin 1400, gpuClkMax 1400)" for GPU 00000009:00:00.0
Warning: persistence mode is disabled on device 00000009:00:00.0. See the Known Issues section of the nvidia-smi(1) man page for more information. Run with [--help | -h] switch to get more information on how to enable persistence mode.
GPU clocks set to "(gpuClkMin 1400, gpuClkMax 1400)" for GPU 0000000A:00:00.0
Warning: persistence mode is disabled on device 0000000A:00:00.0. See the Known Issues section of the nvidia-smi(1) man page for more information. Run with [--help | -h] switch to get more information on how to enable persistence mode.
GPU clocks set to "(gpuClkMin 1400, gpuClkMax 1400)" for GPU 0000000B:00:00.0
Warning: persistence mode is disabled on device 0000000B:00:00.0. See the Known Issues section of the nvidia-smi(1) man page for more information. Run with [--help | -h] switch to get more information on how to enable persistence mode.
GPU clocks set to "(gpuClkMin 1400, gpuClkMax 1400)" for GPU 0000000C:00:00.0
Warning: persistence mode is disabled on device 0000000C:00:00.0. See the Known Issues section of the nvidia-smi(1) man page for more information. Run with [--help | -h] switch to get more information on how to enable persistence mode.
All done.
GPU clocks set to "(gpuClkMin 1600, gpuClkMax 1600)" for GPU 00000001:00:00.0
Warning: persistence mode is disabled on device 00000001:00:00.0. See the Known Issues section of the nvidia-smi(1) man page for more information. Run with [--help | -h] switch to get more information on how to enable persistence mode.
GPU clocks set to "(gpuClkMin 1600, gpuClkMax 1600)" for GPU 00000002:00:00.0
Warning: persistence mode is disabled on device 00000002:00:00.0. See the Known Issues section of the nvidia-smi(1) man page for more information. Run with [--help | -h] switch to get more information on how to enable persistence mode.
GPU clocks set to "(gpuClkMin 1600, gpuClkMax 1600)" for GPU 00000003:00:00.0
Warning: persistence mode is disabled on device 00000003:00:00.0. See the Known Issues section of the nvidia-smi(1) man page for more information. Run with [--help | -h] switch to get more information on how to enable persistence mode.
GPU clocks set to "(gpuClkMin 1600, gpuClkMax 1600)" for GPU 00000008:00:00.0
Warning: persistence mode is disabled on device 00000008:00:00.0. See the Known Issues section of the nvidia-smi(1) man page for more information. Run with [--help | -h] switch to get more information on how to enable persistence mode.
GPU clocks set to "(gpuClkMin 1600, gpuClkMax 1600)" for GPU 00000009:00:00.0
Warning: persistence mode is disabled on device 00000009:00:00.0. See the Known Issues section of the nvidia-smi(1) man page for more information. Run with [--help | -h] switch to get more information on how to enable persistence mode.
GPU clocks set to "(gpuClkMin 1600, gpuClkMax 1600)" for GPU 0000000A:00:00.0
Warning: persistence mode is disabled on device 0000000A:00:00.0. See the Known Issues section of the nvidia-smi(1) man page for more information. Run with [--help | -h] switch to get more information on how to enable persistence mode.
GPU clocks set to "(gpuClkMin 1600, gpuClkMax 1600)" for GPU 0000000B:00:00.0
Warning: persistence mode is disabled on device 0000000B:00:00.0. See the Known Issues section of the nvidia-smi(1) man page for more information. Run with [--help | -h] switch to get more information on how to enable persistence mode.
GPU clocks set to "(gpuClkMin 1600, gpuClkMax 1600)" for GPU 0000000C:00:00.0
Warning: persistence mode is disabled on device 0000000C:00:00.0. See the Known Issues section of the nvidia-smi(1) man page for more information. Run with [--help | -h] switch to get more information on how to enable persistence mode.
All done.
P50 ttft = 0.08546137809753418
P99 ttft = 0.14922530174255372
Average tbt = 0.04242111624103708
P50 tbt = 0.030710220336914062
P99 tbt = 0.0940341033935547
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.0987135645854904
P50 ttft = 0.09098291397094727
P99 ttft = 0.158946304321289
Average tbt = 0.055535970251244335
P50 tbt = 0.031882095336914065
P99 tbt = 0.09539595603942873
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.0947655439376831
P50 ttft = 0.08550739288330078
P99 ttft = 0.15763777494430542
Average tbt = 0.06789863464680133
P50 tbt = 0.0702487707138067
P99 tbt = 0.0958969955444336
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.10623959586733864
P50 ttft = 0.09175539016723633
P99 ttft = 0.21615765571594217
Average tbt = 0.08483402797154026
P50 tbt = 0.09677910804748535
P99 tbt = 0.09804458236694336
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.10260956375687211
P50 ttft = 0.09110212326049805
P99 ttft = 0.17613197326660154
Average tbt = 0.08812052320550992
P50 tbt = 0.09739522933959963
P99 tbt = 0.0982069878578186
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.10925002859420135
P50 ttft = 0.09626960754394531
P99 ttft = 0.21573832988739008
Average tbt = 0.09062815634142452
P50 tbt = 0.09864082336425782
P99 tbt = 0.10284025287628175
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.11736564776476692
P50 ttft = 0.10182058811187744
P99 ttft = 0.25656054019928004
Average tbt = 0.09374356375021102
P50 tbt = 0.10387554168701171
P99 tbt = 0.1380994057655339
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.12406673293182816
P50 ttft = 0.10493767261505127
P99 ttft = 0.21667409658432008
Average tbt = 0.09837648557580041
P50 tbt = 0.10466630458831788
P99 tbt = 0.16369912862777708
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.07029940684636433
P50 ttft = 0.06297445297241211
P99 ttft = 0.11061082839965822
Average tbt = 0.02435167630513509
P50 tbt = 0.024387097358703612
P99 tbt = 0.02461145305633545
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.07312507856459845
P50 ttft = 0.07194852828979492
P99 ttft = 0.11131281852722169
Average tbt = 0.02466929072425479
P50 tbt = 0.024584293365478516
P99 tbt = 0.02518411636352539
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.07808253424508231
P50 ttft = 0.07438969612121582
P99 ttft = 0.12134862422943113
Average tbt = 0.025431867327008927
P50 tbt = 0.02542557716369629
P99 tbt = 0.025928996086120606
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.07962136733822706
P50 ttft = 0.07367181777954102
P99 ttft = 0.12500996589660646
Average tbt = 0.02591457715848598
P50 tbt = 0.025781011581420897
P99 tbt = 0.027041406631469728
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.08159855365753174
P50 ttft = 0.0795823335647583
P99 ttft = 0.12442995309829712
Average tbt = 0.026501527786254888
P50 tbt = 0.026569843292236328
P99 tbt = 0.02709942865371704
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.08434683457016945
P50 ttft = 0.07850909233093262
P99 ttft = 0.12902367830276487
Average tbt = 0.028287771344184875
P50 tbt = 0.02683568000793457
P99 tbt = 0.06833864927291894
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.08274649267327296
P50 ttft = 0.07802081108093262
P99 ttft = 0.1344045925140381
Average tbt = 0.035415260759118536
P50 tbt = 0.027258110046386723
P99 tbt = 0.0838994369506836
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.08368633166853204
P50 ttft = 0.07691478729248047
P99 ttft = 0.13449367523193345
Average tbt = 0.043723600456513634
P50 tbt = 0.029028892517089844
P99 tbt = 0.08505506229400635
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.08661513125642817
P50 ttft = 0.07994890213012695
P99 ttft = 0.13777965784072863
Average tbt = 0.0601486895946747
P50 tbt = 0.06313264369964647
P99 tbt = 0.08656213092803955
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.0872588838849749
P50 ttft = 0.08060026168823242
P99 ttft = 0.14188941955566403
Average tbt = 0.0726139572688512
P50 tbt = 0.08585834503173828
P99 tbt = 0.08738597106933593
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.09021722828900372
P50 ttft = 0.08319103717803955
P99 ttft = 0.16258305072784424
Average tbt = 0.07965379909232811
P50 tbt = 0.08723530769348145
P99 tbt = 0.08834942293167114
96
256
600
96
256
600
96
256
600
467 - 96
Average ttft = 0.0935134086288324
P50 ttft = 0.08334946632385254
P99 ttft = 0.17414652824401847
Average tbt = 0.0776192464748351
P50 tbt = 0.08782825469970704
P99 tbt = 0.08907813167572022
96
256
600
96
256
600
96
256