Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added Arrow Interop Benchmarks #17194

Open
wants to merge 15 commits into
base: branch-24.12
Choose a base branch
from
Open

Added Arrow Interop Benchmarks #17194

wants to merge 15 commits into from
+245 −0

Conversation

lamarrr
Copy link
Contributor

@lamarrr lamarrr commented Oct 28, 2024 •
edited
Loading

Description

Follows up on #17104 (comment)

Benchmark Results

to_arrow_host

[0] NVIDIA RTX A6000

data_type num_rows num_elements Samples CPU Time Noise GPU Time Noise Elem/s GlobalMem BW BWUtil
INT8 10000 10000 29072x 21.252 us 41.97% 17.205 us 23.84% 581.218M 1.308 GB/s 0.17%
INT8 100000 100000 16064x 35.133 us 25.61% 31.144 us 14.02% 3.211G 7.224 GB/s 0.94%
INT8 1000000 1000000 3184x 164.369 us 17.70% 157.811 us 8.97% 6.337G 14.258 GB/s 1.86%
INT8 10000000 10000000 1328x 959.428 us 1.91% 955.106 us 1.59% 10.470G 23.558 GB/s 3.07%
INT16 10000 10000 27360x 22.232 us 36.99% 18.284 us 23.73% 546.915M 2.324 GB/s 0.30%
INT16 100000 100000 10528x 51.646 us 18.93% 47.550 us 11.19% 2.103G 8.938 GB/s 1.16%
INT16 1000000 1000000 2208x 231.548 us 4.83% 227.602 us 2.93% 4.394G 18.673 GB/s 2.43%
INT16 10000000 10000000 2544x 1.728 ms 6.91% 1.724 ms 6.89% 5.799G 24.646 GB/s 3.21%
INT32 10000 10000 20384x 28.487 us 24.66% 24.548 us 9.28% 407.361M 3.361 GB/s 0.44%
INT32 100000 100000 7536x 70.436 us 13.77% 66.421 us 7.30% 1.506G 12.421 GB/s 1.62%
INT32 1000000 1000000 1344x 380.333 us 3.03% 376.231 us 2.00% 2.658G 21.928 GB/s 2.85%
INT32 10000000 10000000 1040x 5.848 ms 1.98% 5.844 ms 1.98% 1.711G 14.117 GB/s 1.84%
INT64 10000 10000 15664x 35.948 us 22.49% 31.923 us 8.37% 313.255M 5.090 GB/s 0.66%
INT64 100000 100000 4416x 117.601 us 9.65% 113.433 us 4.26% 881.576M 14.326 GB/s 1.87%
INT64 1000000 1000000 1440x 676.513 us 2.07% 672.437 us 1.75% 1.487G 24.166 GB/s 3.15%
INT64 10000000 10000000 1232x 11.185 ms 1.52% 11.181 ms 1.52% 894.405M 14.534 GB/s 1.89%
UINT8 10000 10000 29488x 20.964 us 42.73% 16.957 us 24.61% 589.713M 1.327 GB/s 0.17%
UINT8 100000 100000 14704x 37.850 us 20.35% 34.030 us 13.28% 2.939G 6.612 GB/s 0.86%
UINT8 1000000 1000000 3216x 160.136 us 5.98% 156.211 us 3.80% 6.402G 14.404 GB/s 1.88%
UINT8 10000000 10000000 624x 960.125 us 2.06% 955.444 us 1.34% 10.466G 23.549 GB/s 3.07%
UINT16 10000 10000 23744x 25.027 us 25.87% 21.066 us 10.55% 474.707M 2.018 GB/s 0.26%
UINT16 100000 100000 10000x 53.917 us 15.90% 50.046 us 10.87% 1.998G 8.492 GB/s 1.11%
UINT16 1000000 1000000 2208x 231.994 us 4.53% 228.041 us 2.81% 4.385G 18.637 GB/s 2.43%
UINT16 10000000 10000000 1408x 1.722 ms 1.48% 1.718 ms 1.36% 5.820G 24.735 GB/s 3.22%
UINT32 10000 10000 20496x 28.409 us 26.99% 24.410 us 10.65% 409.666M 3.380 GB/s 0.44%
UINT32 100000 100000 7808x 68.040 us 11.97% 64.104 us 6.66% 1.560G 12.870 GB/s 1.68%
UINT32 1000000 1000000 1344x 380.250 us 3.74% 375.999 us 2.32% 2.660G 21.942 GB/s 2.86%
UINT32 10000000 10000000 960x 5.834 ms 1.47% 5.830 ms 1.45% 1.715G 14.152 GB/s 1.84%
UINT64 10000 10000 17264x 32.957 us 25.61% 28.977 us 15.44% 345.097M 5.608 GB/s 0.73%
UINT64 100000 100000 4320x 120.012 us 8.77% 115.966 us 4.49% 862.320M 14.013 GB/s 1.82%
UINT64 1000000 1000000 848x 677.335 us 2.67% 672.809 us 1.72% 1.486G 24.152 GB/s 3.14%
UINT64 10000000 10000000 576x 11.182 ms 1.13% 11.177 ms 1.12% 894.678M 14.539 GB/s 1.89%
FLOAT32 10000 10000 20256x 28.715 us 27.08% 24.691 us 10.12% 405.010M 3.341 GB/s 0.44%
FLOAT32 100000 100000 7568x 70.017 us 13.17% 66.095 us 7.87% 1.513G 12.482 GB/s 1.63%
FLOAT32 1000000 1000000 1344x 380.154 us 3.99% 375.810 us 2.20% 2.661G 21.953 GB/s 2.86%
FLOAT32 10000000 10000000 2556x 5.854 ms 3.23% 5.850 ms 3.23% 1.709G 14.102 GB/s 1.84%
FLOAT64 10000 10000 15952x 35.341 us 21.00% 31.359 us 8.88% 318.892M 5.182 GB/s 0.67%
FLOAT64 100000 100000 4384x 118.503 us 8.76% 114.446 us 4.43% 873.773M 14.199 GB/s 1.85%
FLOAT64 1000000 1000000 800x 675.926 us 2.09% 671.826 us 1.49% 1.488G 24.188 GB/s 3.15%
FLOAT64 10000000 10000000 1338x 11.198 ms 1.52% 11.194 ms 1.52% 893.335M 14.517 GB/s 1.89%
BOOL8 10000 10000 15376x 36.642 us 31.61% 32.527 us 20.68% 307.438M 691.735 MB/s 0.09%
BOOL8 100000 100000 13984x 39.670 us 24.86% 35.785 us 16.98% 2.794G 6.288 GB/s 0.82%
BOOL8 1000000 1000000 7344x 72.331 us 16.47% 68.215 us 12.26% 14.660G 32.984 GB/s 4.29%
BOOL8 10000000 10000000 2240x 404.741 us 4.37% 400.347 us 2.96% 24.978G 56.201 GB/s 7.32%
TIMESTAMP_SECONDS 10000 10000 15856x 35.529 us 20.03% 31.546 us 7.57% 316.998M 5.151 GB/s 0.67%
TIMESTAMP_SECONDS 100000 100000 4272x 121.361 us 8.26% 117.293 us 4.77% 852.565M 13.854 GB/s 1.80%
TIMESTAMP_SECONDS 1000000 1000000 784x 676.258 us 2.55% 671.716 us 1.54% 1.489G 24.192 GB/s 3.15%
TIMESTAMP_SECONDS 10000000 10000000 1341x 11.177 ms 1.41% 11.173 ms 1.41% 895.045M 14.544 GB/s 1.89%
TIMESTAMP_MILLISECONDS 10000 10000 15856x 35.579 us 22.49% 31.545 us 7.89% 317.003M 5.151 GB/s 0.67%
TIMESTAMP_MILLISECONDS 100000 100000 4416x 117.302 us 7.98% 113.290 us 4.20% 882.692M 14.344 GB/s 1.87%
TIMESTAMP_MILLISECONDS 1000000 1000000 864x 674.798 us 2.12% 670.543 us 1.29% 1.491G 24.234 GB/s 3.16%
TIMESTAMP_MILLISECONDS 10000000 10000000 576x 11.157 ms 1.23% 11.153 ms 1.21% 896.613M 14.570 GB/s 1.90%
TIMESTAMP_MICROSECONDS 10000 10000 17504x 32.545 us 25.28% 28.567 us 15.80% 350.051M 5.688 GB/s 0.74%
TIMESTAMP_MICROSECONDS 100000 100000 4336x 119.435 us 7.30% 115.471 us 4.39% 866.017M 14.073 GB/s 1.83%
TIMESTAMP_MICROSECONDS 1000000 1000000 1520x 676.247 us 2.39% 672.027 us 1.83% 1.488G 24.181 GB/s 3.15%
TIMESTAMP_MICROSECONDS 10000000 10000000 816x 11.156 ms 1.20% 11.151 ms 1.20% 896.750M 14.572 GB/s 1.90%
TIMESTAMP_NANOSECONDS 10000 10000 16192x 34.879 us 22.80% 30.903 us 8.13% 323.595M 5.258 GB/s 0.68%
TIMESTAMP_NANOSECONDS 100000 100000 4336x 119.612 us 9.56% 115.532 us 4.33% 865.558M 14.065 GB/s 1.83%
TIMESTAMP_NANOSECONDS 1000000 1000000 752x 675.138 us 2.14% 671.042 us 1.49% 1.490G 24.216 GB/s 3.15%
TIMESTAMP_NANOSECONDS 10000000 10000000 496x 11.180 ms 1.16% 11.177 ms 1.16% 894.732M 14.539 GB/s 1.89%
DURATION_SECONDS 10000 10000 15984x 35.291 us 19.43% 31.309 us 7.47% 319.398M 5.190 GB/s 0.68%
DURATION_SECONDS 100000 100000 4432x 117.139 us 10.01% 113.039 us 4.12% 884.651M 14.376 GB/s 1.87%
DURATION_SECONDS 1000000 1000000 848x 676.219 us 2.39% 671.922 us 1.77% 1.488G 24.184 GB/s 3.15%
DURATION_SECONDS 10000000 10000000 1340x 11.180 ms 1.80% 11.176 ms 1.78% 894.745M 14.540 GB/s 1.89%
DURATION_MILLISECONDS 10000 10000 17328x 32.822 us 24.28% 28.874 us 15.03% 346.337M 5.628 GB/s 0.73%
DURATION_MILLISECONDS 100000 100000 4304x 120.326 us 7.39% 116.388 us 4.21% 859.195M 13.962 GB/s 1.82%
DURATION_MILLISECONDS 1000000 1000000 784x 675.464 us 2.17% 671.278 us 1.63% 1.490G 24.208 GB/s 3.15%
DURATION_MILLISECONDS 10000000 10000000 896x 11.185 ms 1.16% 11.181 ms 1.15% 894.344M 14.533 GB/s 1.89%
DURATION_MICROSECONDS 10000 10000 16112x 35.011 us 19.97% 31.047 us 7.62% 322.097M 5.234 GB/s 0.68%
DURATION_MICROSECONDS 100000 100000 4304x 120.436 us 7.84% 116.428 us 4.38% 858.898M 13.957 GB/s 1.82%
DURATION_MICROSECONDS 1000000 1000000 768x 675.741 us 2.31% 671.433 us 1.55% 1.489G 24.202 GB/s 3.15%
DURATION_MICROSECONDS 10000000 10000000 1340x 11.183 ms 1.38% 11.179 ms 1.38% 894.559M 14.537 GB/s 1.89%
DURATION_NANOSECONDS 10000 10000 16016x 35.242 us 20.95% 31.247 us 8.49% 320.028M 5.200 GB/s 0.68%
DURATION_NANOSECONDS 100000 100000 4368x 118.709 us 9.19% 114.585 us 4.21% 872.718M 14.182 GB/s 1.85%
DURATION_NANOSECONDS 1000000 1000000 1200x 675.703 us 2.18% 671.512 us 1.44% 1.489G 24.199 GB/s 3.15%
DURATION_NANOSECONDS 10000000 10000000 592x 11.162 ms 1.24% 11.158 ms 1.24% 896.200M 14.563 GB/s 1.90%
STRING 10000 10000 10304x 52.685 us 19.95% 48.550 us 10.43% 205.972M 8.176 GB/s 1.06%
STRING 100000 100000 2144x 238.538 us 5.89% 234.160 us 2.95% 427.058M 16.914 GB/s 2.20%
STRING 1000000 1000000 768x 1.619 ms 1.07% 1.615 ms 1.05% 619.028M 24.520 GB/s 3.19%
STRING 10000000 10000000 304x 27.231 ms 1.01% 27.226 ms 1.00% 367.291M 14.558 GB/s 1.90%
LIST 10000 10000 832x 1.033 ms 1.92% 1.028 ms 1.27% 9.726M 23.090 GB/s 3.01%
LIST 100000 100000 939x 15.958 ms 1.92% 15.954 ms 1.92% 6.268M 14.929 GB/s 1.94%
LIST 1000000 1000000 94x 160.490 ms 1.13% 160.485 ms 1.13% 6.231M 14.839 GB/s 1.93%
DECIMAL128 10000 10000 12192x 44.887 us 21.96% 41.045 us 11.83% 243.633M 7.857 GB/s 1.02%
DECIMAL128 100000 100000 2720x 190.323 us 14.07% 183.901 us 3.17% 543.772M 17.537 GB/s 2.28%
DECIMAL128 1000000 1000000 896x 1.288 ms 2.50% 1.281 ms 1.09% 780.619M 25.175 GB/s 3.28%
DECIMAL128 10000000 10000000 560x 21.822 ms 1.14% 21.818 ms 1.14% 458.339M 14.781 GB/s 1.92%
STRUCT 10000 10000 2992x 171.285 us 9.49% 167.434 us 8.49% 59.725M 3.380 GB/s 0.44%
STRUCT 100000 100000 2672x 454.446 us 4.12% 450.347 us 3.73% 222.051M 12.556 GB/s 1.63%
STRUCT 1000000 1000000 1248x 2.488 ms 1.39% 2.484 ms 1.23% 402.629M 22.770 GB/s 2.96%
STRUCT 10000000 10000000 384x 39.069 ms 1.23% 39.065 ms 1.23% 255.980M 14.482 GB/s 1.89%

to_arrow_device

[0] NVIDIA RTX A6000

data_type num_rows num_elements Samples CPU Time Noise GPU Time Noise Elem/s GlobalMem BW BWUtil
INT8 10000 10000 187552x 6.406 us 168.52% 2.666 us 34.95% 3.751G 8.439 GB/s 1.10%
INT8 100000 100000 187168x 6.372 us 173.78% 2.672 us 44.74% 37.431G 84.221 GB/s 10.96%
INT8 1000000 1000000 187888x 6.405 us 177.43% 2.661 us 29.95% 375.750G 845.439 GB/s 110.07%
INT8 10000000 10000000 187728x 6.415 us 171.40% 2.664 us 29.86% 3.754T 8.447 TB/s 1099.78%
INT16 10000 10000 187808x 6.417 us 178.60% 2.662 us 28.70% 3.756G 15.963 GB/s 2.08%
INT16 100000 100000 187280x 6.358 us 168.88% 2.670 us 36.79% 37.454G 159.180 GB/s 20.72%
INT16 1000000 1000000 187728x 6.395 us 176.29% 2.663 us 41.67% 375.446G 1.596 TB/s 207.74%
INT16 10000000 10000000 187456x 6.403 us 165.17% 2.667 us 42.47% 3.749T 15.933 TB/s 2074.41%
INT32 10000 10000 188016x 6.398 us 172.65% 2.659 us 33.23% 3.760G 31.023 GB/s 4.04%
INT32 100000 100000 187488x 6.359 us 178.43% 2.667 us 35.97% 37.494G 309.328 GB/s 40.27%
INT32 1000000 1000000 187200x 6.402 us 177.64% 2.671 us 38.64% 374.396G 3.089 TB/s 402.13%
INT32 10000000 10000000 187520x 6.396 us 169.53% 2.666 us 34.26% 3.750T 30.940 TB/s 4028.18%
INT64 10000 10000 187728x 6.420 us 182.10% 2.664 us 29.47% 3.754G 61.007 GB/s 7.94%
INT64 100000 100000 187328x 6.348 us 171.47% 2.669 us 66.45% 37.463G 608.773 GB/s 79.26%
INT64 1000000 1000000 187696x 6.387 us 170.83% 2.664 us 38.85% 375.386G 6.100 TB/s 794.17%
INT64 10000000 10000000 187408x 6.401 us 170.05% 2.668 us 43.54% 3.748T 60.903 TB/s 7929.14%
UINT8 10000 10000 187952x 6.401 us 171.88% 2.660 us 28.51% 3.759G 8.458 GB/s 1.10%
UINT8 100000 100000 187712x 6.352 us 163.91% 2.664 us 28.78% 37.542G 84.469 GB/s 11.00%
UINT8 1000000 1000000 187456x 6.422 us 200.63% 2.667 us 35.95% 374.891G 843.504 GB/s 109.82%
UINT8 10000000 10000000 187568x 6.411 us 178.05% 2.666 us 37.99% 3.751T 8.440 TB/s 1098.85%
UINT16 10000 10000 187664x 6.417 us 168.75% 2.664 us 35.09% 3.753G 15.951 GB/s 2.08%
UINT16 100000 100000 187824x 6.341 us 168.51% 2.662 us 29.23% 37.563G 159.643 GB/s 20.78%
UINT16 1000000 1000000 187488x 6.414 us 175.86% 2.667 us 37.58% 374.961G 1.594 TB/s 207.47%
UINT16 10000000 10000000 187520x 6.406 us 172.13% 2.666 us 30.12% 3.750T 15.939 TB/s 2075.13%
UINT32 10000 10000 187760x 6.418 us 185.42% 2.663 us 30.52% 3.755G 30.980 GB/s 4.03%
UINT32 100000 100000 187488x 6.351 us 172.60% 2.667 us 35.84% 37.497G 309.352 GB/s 40.28%
UINT32 1000000 1000000 187408x 6.398 us 171.30% 2.668 us 33.46% 374.808G 3.092 TB/s 402.58%
UINT32 10000000 10000000 187584x 6.397 us 168.92% 2.666 us 30.18% 3.752T 30.951 TB/s 4029.53%
UINT64 10000 10000 187568x 6.424 us 180.61% 2.666 us 38.59% 3.751G 60.955 GB/s 7.94%
UINT64 100000 100000 187856x 6.353 us 171.61% 2.662 us 28.45% 37.569G 610.502 GB/s 79.48%
UINT64 1000000 1000000 187376x 6.396 us 174.28% 2.668 us 36.08% 374.744G 6.090 TB/s 792.82%
UINT64 10000000 10000000 187456x 6.409 us 170.61% 2.667 us 30.71% 3.749T 60.919 TB/s 7931.13%
FLOAT32 10000 10000 187728x 6.417 us 178.33% 2.664 us 29.33% 3.754G 30.974 GB/s 4.03%
FLOAT32 100000 100000 187568x 6.352 us 167.60% 2.666 us 37.68% 37.511G 309.465 GB/s 40.29%
FLOAT32 1000000 1000000 187792x 6.402 us 179.71% 2.663 us 29.98% 375.565G 3.098 TB/s 403.39%
FLOAT32 10000000 10000000 187392x 6.410 us 170.31% 2.668 us 49.34% 3.748T 30.919 TB/s 4025.39%
FLOAT64 10000 10000 187520x 6.432 us 177.20% 2.667 us 30.30% 3.750G 60.939 GB/s 7.93%
FLOAT64 100000 100000 187664x 6.347 us 173.04% 2.665 us 39.57% 37.530G 609.870 GB/s 79.40%
FLOAT64 1000000 1000000 187552x 6.375 us 174.43% 2.666 us 34.32% 375.093G 6.095 TB/s 793.55%
FLOAT64 10000000 10000000 187632x 6.400 us 167.20% 2.665 us 37.00% 3.752T 60.975 TB/s 7938.51%
BOOL8 10000 10000 24368x 24.575 us 32.08% 20.528 us 13.35% 487.151M 1.096 GB/s 0.14%
BOOL8 100000 100000 23712x 25.046 us 34.78% 21.092 us 17.57% 4.741G 10.668 GB/s 1.39%
BOOL8 1000000 1000000 17696x 32.264 us 23.62% 28.270 us 11.49% 35.373G 79.589 GB/s 10.36%
BOOL8 10000000 10000000 4816x 107.983 us 14.18% 103.838 us 7.06% 96.304G 216.683 GB/s 28.21%
TIMESTAMP_SECONDS 10000 10000 187696x 6.413 us 179.50% 2.664 us 36.00% 3.754G 60.997 GB/s 7.94%
TIMESTAMP_SECONDS 100000 100000 187456x 6.351 us 173.45% 2.667 us 34.39% 37.491G 609.223 GB/s 79.32%
TIMESTAMP_SECONDS 1000000 1000000 187120x 6.461 us 228.29% 2.672 us 42.50% 374.221G 6.081 TB/s 791.71%
TIMESTAMP_SECONDS 10000000 10000000 185568x 6.734 us 358.19% 2.694 us 61.83% 3.711T 60.308 TB/s 7851.63%
TIMESTAMP_MILLISECONDS 10000 10000 187936x 6.411 us 176.95% 2.661 us 29.71% 3.759G 61.076 GB/s 7.95%
TIMESTAMP_MILLISECONDS 100000 100000 187504x 6.351 us 164.46% 2.667 us 30.84% 37.500G 609.378 GB/s 79.34%
TIMESTAMP_MILLISECONDS 1000000 1000000 187344x 6.411 us 177.54% 2.669 us 34.59% 374.675G 6.088 TB/s 792.67%
TIMESTAMP_MILLISECONDS 10000000 10000000 187792x 6.406 us 172.21% 2.663 us 34.05% 3.756T 61.032 TB/s 7945.88%
TIMESTAMP_MICROSECONDS 10000 10000 187280x 6.450 us 208.50% 2.670 us 42.93% 3.745G 60.864 GB/s 7.92%
TIMESTAMP_MICROSECONDS 100000 100000 187376x 6.354 us 173.46% 2.669 us 31.80% 37.473G 608.939 GB/s 79.28%
TIMESTAMP_MICROSECONDS 1000000 1000000 187328x 6.399 us 168.69% 2.669 us 33.51% 374.630G 6.088 TB/s 792.58%
TIMESTAMP_MICROSECONDS 10000000 10000000 187616x 6.399 us 168.71% 2.665 us 30.99% 3.752T 60.972 TB/s 7938.12%
TIMESTAMP_NANOSECONDS 10000 10000 187216x 6.416 us 165.66% 2.671 us 40.24% 3.744G 60.844 GB/s 7.92%
TIMESTAMP_NANOSECONDS 100000 100000 187360x 6.348 us 163.38% 2.669 us 30.85% 37.470G 608.894 GB/s 79.27%
TIMESTAMP_NANOSECONDS 1000000 1000000 187456x 6.410 us 180.14% 2.667 us 31.55% 374.906G 6.092 TB/s 793.16%
TIMESTAMP_NANOSECONDS 10000000 10000000 187088x 6.432 us 186.56% 2.673 us 32.80% 3.742T 60.803 TB/s 7916.05%
DURATION_SECONDS 10000 10000 187200x 6.414 us 166.32% 2.671 us 35.92% 3.744G 60.839 GB/s 7.92%
DURATION_SECONDS 100000 100000 187488x 6.349 us 168.41% 2.667 us 40.63% 37.495G 609.293 GB/s 79.33%
DURATION_SECONDS 1000000 1000000 187936x 6.386 us 165.90% 2.661 us 29.04% 375.867G 6.108 TB/s 795.19%
DURATION_SECONDS 10000000 10000000 187728x 6.392 us 169.20% 2.664 us 29.30% 3.754T 61.009 TB/s 7942.88%
DURATION_MILLISECONDS 10000 10000 187728x 6.419 us 169.47% 2.664 us 28.68% 3.754G 61.009 GB/s 7.94%
DURATION_MILLISECONDS 100000 100000 187488x 6.357 us 172.20% 2.667 us 28.52% 37.497G 609.324 GB/s 79.33%
DURATION_MILLISECONDS 1000000 1000000 187248x 6.404 us 170.34% 2.670 us 29.16% 374.478G 6.085 TB/s 792.25%
DURATION_MILLISECONDS 10000000 10000000 187632x 6.409 us 173.04% 2.665 us 28.76% 3.752T 60.975 TB/s 7938.51%
DURATION_MICROSECONDS 10000 10000 187296x 6.438 us 202.91% 2.670 us 30.00% 3.746G 60.871 GB/s 7.92%
DURATION_MICROSECONDS 100000 100000 187344x 6.366 us 192.20% 2.669 us 34.84% 37.467G 608.835 GB/s 79.27%
DURATION_MICROSECONDS 1000000 1000000 187584x 6.404 us 173.74% 2.666 us 29.64% 375.142G 6.096 TB/s 793.66%
DURATION_MICROSECONDS 10000000 10000000 187376x 6.412 us 175.46% 2.669 us 56.60% 3.747T 60.892 TB/s 7927.68%
DURATION_NANOSECONDS 10000 10000 187424x 6.410 us 167.43% 2.668 us 44.88% 3.748G 60.912 GB/s 7.93%
DURATION_NANOSECONDS 100000 100000 187680x 6.351 us 168.96% 2.664 us 29.28% 37.535G 609.951 GB/s 79.41%
DURATION_NANOSECONDS 1000000 1000000 187904x 6.406 us 183.12% 2.661 us 33.36% 375.792G 6.107 TB/s 795.03%
DURATION_NANOSECONDS 10000000 10000000 187680x 6.403 us 168.45% 2.664 us 36.60% 3.753T 60.993 TB/s 7940.80%
STRING 10000 10000 66736x 11.516 us 73.99% 7.493 us 22.40% 1.335G 52.974 GB/s 6.90%
STRING 100000 100000 67952x 11.295 us 68.99% 7.359 us 19.28% 13.589G 538.198 GB/s 70.07%
STRING 1000000 1000000 66144x 11.539 us 68.23% 7.561 us 25.85% 132.261G 5.239 TB/s 682.07%
STRING 10000000 10000000 67216x 11.425 us 69.33% 7.439 us 20.83% 1.344T 53.280 TB/s 6936.61%
LIST 10000 10000 187264x 6.430 us 182.94% 2.670 us 47.14% 3.745G 8.891 TB/s 1157.51%
LIST 100000 100000 186992x 6.368 us 170.32% 2.674 us 37.34% 37.398G 89.074 TB/s 11596.70%
LIST 1000000 1000000 187312x 6.390 us 156.56% 2.669 us 34.79% 374.622G 892.125 TB/s 116147.61%
DECIMAL128 10000 10000 186416x 6.536 us 244.76% 2.682 us 42.18% 3.728G 120.236 GB/s 15.65%
DECIMAL128 100000 100000 187552x 6.401 us 165.63% 2.666 us 31.24% 37.508G 1.210 TB/s 157.49%
DECIMAL128 1000000 1000000 187712x 6.347 us 167.84% 2.664 us 28.97% 375.419G 12.107 TB/s 1576.27%
DECIMAL128 10000000 10000000 187312x 6.414 us 188.29% 2.669 us 43.10% 3.746T 120.811 TB/s 15728.70%
STRUCT 10000 10000 59376x 12.413 us 62.14% 8.423 us 14.72% 1.187G 67.198 GB/s 8.75%
STRUCT 100000 100000 59536x 12.414 us 62.54% 8.400 us 13.82% 11.904G 673.108 GB/s 87.63%
STRUCT 1000000 1000000 58736x 12.456 us 62.62% 8.513 us 14.66% 117.466G 6.643 TB/s 864.88%
STRUCT 10000000 10000000 59280x 12.418 us 62.10% 8.436 us 15.10% 1.185T 67.064 TB/s 8731.18%

from_arrow_host

[0] NVIDIA RTX A6000

data_type num_rows num_elements Samples CPU Time Noise GPU Time Noise Elem/s GlobalMem BW BWUtil
INT8 10000 10000 139344x 7.758 us 142.64% 3.588 us 33.63% 2.787G 6.270 GB/s 0.82%
INT8 100000 100000 29296x 20.972 us 37.89% 17.076 us 20.01% 5.856G 13.177 GB/s 1.72%
INT8 1000000 1000000 4352x 119.471 us 7.15% 115.256 us 2.59% 8.676G 19.522 GB/s 2.54%
INT8 10000000 10000000 928x 973.354 us 1.61% 968.331 us 0.86% 10.327G 23.236 GB/s 3.03%
INT16 10000 10000 109920x 8.469 us 104.91% 4.549 us 30.80% 2.198G 9.342 GB/s 1.22%
INT16 100000 100000 15552x 36.048 us 25.78% 32.169 us 16.15% 3.109G 13.212 GB/s 1.72%
INT16 1000000 1000000 2496x 204.820 us 5.43% 200.376 us 1.63% 4.991G 21.210 GB/s 2.76%
INT16 10000000 10000000 912x 1.830 ms 0.88% 1.826 ms 0.79% 5.476G 23.272 GB/s 3.03%
INT32 10000 10000 76240x 10.349 us 72.35% 6.559 us 16.46% 1.525G 12.579 GB/s 1.64%
INT32 100000 100000 10208x 52.906 us 15.38% 48.988 us 11.78% 2.041G 16.841 GB/s 2.19%
INT32 1000000 1000000 1808x 372.205 us 3.20% 367.629 us 0.90% 2.720G 22.441 GB/s 2.92%
INT32 10000000 10000000 848x 3.563 ms 0.76% 3.558 ms 0.72% 2.811G 23.187 GB/s 3.02%
INT64 10000 10000 36528x 17.884 us 51.93% 13.694 us 26.84% 730.247M 11.867 GB/s 1.54%
INT64 100000 100000 5616x 93.093 us 12.71% 89.103 us 9.26% 1.122G 18.237 GB/s 2.37%
INT64 1000000 1000000 1456x 707.896 us 1.39% 703.540 us 0.85% 1.421G 23.097 GB/s 3.01%
INT64 10000000 10000000 944x 6.954 ms 0.60% 6.949 ms 0.59% 1.439G 23.384 GB/s 3.04%
UINT8 10000 10000 139664x 7.765 us 135.04% 3.580 us 29.69% 2.793G 6.285 GB/s 0.82%
UINT8 100000 100000 24688x 24.080 us 37.05% 20.261 us 28.66% 4.936G 11.105 GB/s 1.45%
UINT8 1000000 1000000 4352x 119.332 us 6.92% 114.989 us 2.63% 8.696G 19.567 GB/s 2.55%
UINT8 10000000 10000000 976x 973.233 us 0.99% 969.112 us 0.89% 10.319G 23.217 GB/s 3.02%
UINT16 10000 10000 109952x 8.512 us 111.07% 4.548 us 22.82% 2.199G 9.345 GB/s 1.22%
UINT16 100000 100000 16432x 34.304 us 26.39% 30.430 us 14.91% 3.286G 13.966 GB/s 1.82%
UINT16 1000000 1000000 2496x 206.137 us 6.52% 201.364 us 2.03% 4.966G 21.106 GB/s 2.75%
UINT16 10000000 10000000 1008x 1.831 ms 1.09% 1.827 ms 0.93% 5.474G 23.266 GB/s 3.03%
UINT32 10000 10000 76112x 10.376 us 78.30% 6.570 us 20.38% 1.522G 12.557 GB/s 1.63%
UINT32 100000 100000 9664x 55.783 us 18.11% 51.756 us 8.81% 1.932G 15.940 GB/s 2.08%
UINT32 1000000 1000000 1360x 373.256 us 3.27% 368.749 us 1.34% 2.712G 22.373 GB/s 2.91%
UINT32 10000000 10000000 816x 3.558 ms 0.77% 3.553 ms 0.69% 2.814G 23.217 GB/s 3.02%
UINT64 10000 10000 37328x 17.602 us 48.25% 13.400 us 24.91% 746.267M 12.127 GB/s 1.58%
UINT64 100000 100000 5712x 91.749 us 13.95% 87.755 us 10.73% 1.140G 18.517 GB/s 2.41%
UINT64 1000000 1000000 1152x 707.866 us 1.65% 703.230 us 0.67% 1.422G 23.108 GB/s 3.01%
UINT64 10000000 10000000 576x 6.958 ms 0.64% 6.954 ms 0.63% 1.438G 23.368 GB/s 3.04%
FLOAT32 10000 10000 76240x 10.370 us 72.49% 6.559 us 17.37% 1.525G 12.579 GB/s 1.64%
FLOAT32 100000 100000 9776x 55.013 us 18.70% 51.168 us 12.66% 1.954G 16.123 GB/s 2.10%
FLOAT32 1000000 1000000 1712x 372.576 us 1.66% 368.550 us 1.22% 2.713G 22.385 GB/s 2.91%
FLOAT32 10000000 10000000 880x 3.558 ms 0.73% 3.553 ms 0.70% 2.814G 23.219 GB/s 3.02%
FLOAT64 10000 10000 33728x 18.574 us 48.64% 14.826 us 22.00% 674.477M 10.960 GB/s 1.43%
FLOAT64 100000 100000 5696x 91.811 us 10.54% 87.879 us 8.91% 1.138G 18.491 GB/s 2.41%
FLOAT64 1000000 1000000 1088x 708.323 us 1.90% 703.541 us 0.78% 1.421G 23.097 GB/s 3.01%
FLOAT64 10000000 10000000 624x 6.952 ms 0.53% 6.947 ms 0.53% 1.439G 23.392 GB/s 3.05%
BOOL8 10000 10000 52304x 13.348 us 64.37% 9.561 us 33.31% 1.046G 2.353 GB/s 0.31%
BOOL8 100000 100000 42304x 15.551 us 55.08% 11.823 us 27.09% 8.458G 19.031 GB/s 2.48%
BOOL8 1000000 1000000 10048x 53.727 us 22.55% 49.797 us 18.26% 20.082G 45.184 GB/s 5.88%
BOOL8 10000000 10000000 3056x 319.848 us 6.49% 315.722 us 5.50% 31.673G 71.265 GB/s 9.28%
TIMESTAMP_SECONDS 10000 10000 31168x 19.931 us 45.24% 16.046 us 25.64% 623.195M 10.127 GB/s 1.32%
TIMESTAMP_SECONDS 100000 100000 5680x 91.974 us 12.21% 88.117 us 10.13% 1.135G 18.441 GB/s 2.40%
TIMESTAMP_SECONDS 1000000 1000000 1696x 707.747 us 1.33% 703.426 us 0.65% 1.422G 23.101 GB/s 3.01%
TIMESTAMP_SECONDS 10000000 10000000 1152x 6.955 ms 0.77% 6.950 ms 0.77% 1.439G 23.381 GB/s 3.04%
TIMESTAMP_MILLISECONDS 10000 10000 33408x 19.183 us 44.23% 14.973 us 21.20% 667.888M 10.853 GB/s 1.41%
TIMESTAMP_MILLISECONDS 100000 100000 5600x 93.228 us 12.03% 89.395 us 9.24% 1.119G 18.178 GB/s 2.37%
TIMESTAMP_MILLISECONDS 1000000 1000000 1536x 708.798 us 1.94% 703.984 us 0.86% 1.420G 23.083 GB/s 3.01%
TIMESTAMP_MILLISECONDS 10000000 10000000 72x 6.952 ms 0.44% 6.948 ms 0.44% 1.439G 23.388 GB/s 3.04%
TIMESTAMP_MICROSECONDS 10000 10000 34992x 18.507 us 51.85% 14.293 us 22.20% 699.665M 11.370 GB/s 1.48%
TIMESTAMP_MICROSECONDS 100000 100000 5408x 96.345 us 11.39% 92.640 us 9.24% 1.079G 17.541 GB/s 2.28%
TIMESTAMP_MICROSECONDS 1000000 1000000 1504x 708.871 us 2.11% 704.299 us 0.83% 1.420G 23.073 GB/s 3.00%
TIMESTAMP_MICROSECONDS 10000000 10000000 544x 6.957 ms 0.66% 6.952 ms 0.65% 1.438G 23.374 GB/s 3.04%
TIMESTAMP_NANOSECONDS 10000 10000 33872x 18.996 us 46.98% 14.767 us 21.32% 677.176M 11.004 GB/s 1.43%
TIMESTAMP_NANOSECONDS 100000 100000 5456x 95.756 us 12.85% 91.892 us 10.73% 1.088G 17.684 GB/s 2.30%
TIMESTAMP_NANOSECONDS 1000000 1000000 1808x 707.606 us 1.16% 703.332 us 0.58% 1.422G 23.104 GB/s 3.01%
TIMESTAMP_NANOSECONDS 10000000 10000000 704x 6.954 ms 0.65% 6.948 ms 0.58% 1.439G 23.389 GB/s 3.05%
DURATION_SECONDS 10000 10000 31488x 19.796 us 38.98% 15.885 us 21.47% 629.529M 10.230 GB/s 1.33%
DURATION_SECONDS 100000 100000 5392x 96.865 us 12.84% 92.982 us 9.60% 1.075G 17.477 GB/s 2.28%
DURATION_SECONDS 1000000 1000000 1120x 709.014 us 2.46% 704.106 us 0.93% 1.420G 23.079 GB/s 3.00%
DURATION_SECONDS 10000000 10000000 1280x 6.953 ms 0.75% 6.949 ms 0.74% 1.439G 23.385 GB/s 3.04%
DURATION_MILLISECONDS 10000 10000 34064x 18.868 us 42.46% 14.679 us 17.74% 681.253M 11.070 GB/s 1.44%
DURATION_MILLISECONDS 100000 100000 5600x 93.270 us 9.89% 89.400 us 7.28% 1.119G 18.177 GB/s 2.37%
DURATION_MILLISECONDS 1000000 1000000 2192x 708.101 us 1.55% 703.667 us 0.77% 1.421G 23.093 GB/s 3.01%
DURATION_MILLISECONDS 10000000 10000000 560x 6.951 ms 0.56% 6.946 ms 0.56% 1.440G 23.394 GB/s 3.05%
DURATION_MICROSECONDS 10000 10000 35056x 18.516 us 51.46% 14.269 us 25.98% 700.839M 11.389 GB/s 1.48%
DURATION_MICROSECONDS 100000 100000 5664x 92.276 us 11.36% 88.522 us 10.47% 1.130G 18.357 GB/s 2.39%
DURATION_MICROSECONDS 1000000 1000000 1536x 708.273 us 1.65% 703.747 us 0.77% 1.421G 23.091 GB/s 3.01%
DURATION_MICROSECONDS 10000000 10000000 688x 6.953 ms 0.58% 6.948 ms 0.57% 1.439G 23.386 GB/s 3.04%
DURATION_NANOSECONDS 10000 10000 33856x 18.968 us 45.73% 14.771 us 23.05% 677.012M 11.001 GB/s 1.43%
DURATION_NANOSECONDS 100000 100000 5648x 92.077 us 11.25% 88.701 us 7.54% 1.127G 18.320 GB/s 2.39%
DURATION_NANOSECONDS 1000000 1000000 1824x 708.725 us 1.66% 704.249 us 1.07% 1.420G 23.074 GB/s 3.00%
DURATION_NANOSECONDS 10000000 10000000 135x 6.953 ms 0.50% 6.949 ms 0.50% 1.439G 23.386 GB/s 3.04%
STRING 10000 10000 20176x 28.961 us 36.16% 24.797 us 21.44% 403.274M 16.008 GB/s 2.08%
STRING 100000 100000 2880x 187.787 us 6.97% 184.029 us 6.11% 543.394M 21.522 GB/s 2.80%
STRING 1000000 1000000 768x 1.690 ms 0.91% 1.686 ms 0.80% 593.081M 23.492 GB/s 3.06%
STRING 10000000 10000000 30x 16.869 ms 0.39% 16.859 ms 0.38% 593.169M 23.512 GB/s 3.06%
LIST 10000 10000 5248x 99.518 us 14.48% 95.441 us 12.60% 104.777M 15.052 GB/s 1.96%
LIST 100000 100000 3056x 646.939 us 2.36% 642.830 us 2.07% 155.562M 22.414 GB/s 2.92%
LIST 1000000 1000000 656x 6.178 ms 0.67% 6.174 ms 0.66% 161.966M 23.347 GB/s 3.04%
LIST 10000000 10000000 11x 61.158 ms 0.37% 61.153 ms 0.37% 163.523M 23.560 GB/s 3.07%
DECIMAL128 10000 10000 22128x 26.802 us 35.40% 22.611 us 20.14% 442.264M 14.263 GB/s 1.86%
DECIMAL128 100000 100000 3024x 169.340 us 6.40% 165.522 us 5.01% 604.150M 19.484 GB/s 2.54%
DECIMAL128 1000000 1000000 1824x 1.380 ms 0.84% 1.376 ms 0.63% 726.701M 23.436 GB/s 3.05%
DECIMAL128 10000000 10000000 37x 13.696 ms 0.41% 13.692 ms 0.40% 730.369M 23.554 GB/s 3.07%
STRUCT 10000 10000 4352x 118.894 us 17.51% 114.903 us 15.68% 87.030M 4.882 GB/s 0.64%
STRUCT 100000 100000 3072x 348.455 us 9.19% 344.487 us 8.88% 290.287M 16.269 GB/s 2.12%
STRUCT 1000000 1000000 1488x 2.522 ms 1.59% 2.518 ms 1.58% 397.120M 22.260 GB/s 2.90%
STRUCT 10000000 10000000 21x 24.168 ms 0.37% 24.164 ms 0.37% 413.838M 23.206 GB/s 3.02%

from_arrow_device

[0] NVIDIA RTX A6000

data_type num_rows num_elements Samples CPU Time Noise GPU Time Noise Elem/s GlobalMem BW BWUtil
INT8 10000 10000 247104x 5.087 us 356.39% 1.318 us 52.86% 7.588G 17.072 GB/s 2.22%
INT8 100000 100000 247359x 5.100 us 339.93% 1.317 us 51.63% 75.937G 170.859 GB/s 22.24%
INT8 1000000 1000000 247275x 5.031 us 358.42% 1.316 us 50.75% 759.705G 1.709 TB/s 222.54%
INT8 10000000 10000000 246951x 5.085 us 353.57% 1.318 us 50.16% 7.589T 17.076 TB/s 2223.12%
INT16 10000 10000 247035x 5.095 us 362.07% 1.321 us 58.06% 7.567G 32.161 GB/s 4.19%
INT16 100000 100000 247220x 5.115 us 358.45% 1.320 us 55.88% 75.739G 321.890 GB/s 41.91%
INT16 1000000 1000000 247266x 5.043 us 336.08% 1.320 us 55.14% 757.428G 3.219 TB/s 419.10%
INT16 10000000 10000000 246722x 5.099 us 383.78% 1.325 us 59.73% 7.549T 32.084 TB/s 4177.07%
INT32 10000 10000 247183x 5.102 us 353.56% 1.321 us 54.09% 7.570G 62.453 GB/s 8.13%
INT32 100000 100000 247178x 5.106 us 343.70% 1.318 us 51.48% 75.901G 626.180 GB/s 81.52%
INT32 1000000 1000000 247357x 5.042 us 386.28% 1.320 us 55.38% 757.457G 6.249 TB/s 813.57%
INT32 10000000 10000000 246982x 5.074 us 341.74% 1.316 us 50.43% 7.600T 62.700 TB/s 8163.06%
INT64 10000 10000 247042x 5.094 us 350.29% 1.324 us 57.42% 7.555G 122.774 GB/s 15.98%
INT64 100000 100000 247152x 5.107 us 356.29% 1.320 us 53.37% 75.742G 1.231 TB/s 160.24%
INT64 1000000 1000000 247238x 5.038 us 345.42% 1.326 us 58.92% 754.244G 12.256 TB/s 1595.69%
INT64 10000000 10000000 246776x 5.086 us 351.54% 1.323 us 55.61% 7.556T 122.790 TB/s 15986.34%
UINT8 10000 10000 247079x 5.102 us 349.76% 1.322 us 55.66% 7.565G 17.022 GB/s 2.22%
UINT8 100000 100000 247051x 5.102 us 351.15% 1.320 us 55.80% 75.729G 170.391 GB/s 22.18%
UINT8 1000000 1000000 247244x 5.026 us 340.20% 1.317 us 53.43% 759.482G 1.709 TB/s 222.48%
UINT8 10000000 10000000 247007x 5.065 us 342.98% 1.317 us 52.36% 7.590T 17.078 TB/s 2223.47%
UINT16 10000 10000 246958x 5.081 us 337.62% 1.316 us 51.27% 7.600G 32.300 GB/s 4.21%
UINT16 100000 100000 246931x 5.116 us 343.21% 1.320 us 54.90% 75.768G 322.015 GB/s 41.92%
UINT16 1000000 1000000 247160x 5.033 us 347.32% 1.315 us 50.44% 760.623G 3.233 TB/s 420.87%
UINT16 10000000 10000000 246887x 5.094 us 346.76% 1.318 us 57.29% 7.585T 32.237 TB/s 4196.95%
UINT32 10000 10000 247196x 5.101 us 363.10% 1.319 us 52.52% 7.579G 62.529 GB/s 8.14%
UINT32 100000 100000 247221x 5.113 us 352.17% 1.320 us 52.19% 75.784G 625.217 GB/s 81.40%
UINT32 1000000 1000000 247335x 5.027 us 344.98% 1.319 us 53.12% 758.062G 6.254 TB/s 814.22%
UINT32 10000000 10000000 246881x 5.082 us 364.64% 1.320 us 53.20% 7.578T 62.516 TB/s 8139.02%
UINT64 10000 10000 246803x 5.102 us 345.88% 1.321 us 54.41% 7.570G 123.016 GB/s 16.02%
UINT64 100000 100000 246964x 5.129 us 346.81% 1.329 us 62.11% 75.246G 1.223 TB/s 159.19%
UINT64 1000000 1000000 247142x 5.041 us 331.73% 1.321 us 53.67% 757.219G 12.305 TB/s 1601.99%
UINT64 10000000 10000000 246851x 5.099 us 346.52% 1.321 us 52.18% 7.570T 123.005 TB/s 16014.29%
FLOAT32 10000 10000 247069x 5.093 us 354.84% 1.320 us 63.38% 7.577G 62.508 GB/s 8.14%
FLOAT32 100000 100000 247244x 5.111 us 341.94% 1.318 us 51.67% 75.855G 625.800 GB/s 81.47%
FLOAT32 1000000 1000000 247228x 5.035 us 331.21% 1.319 us 54.98% 757.876G 6.252 TB/s 814.02%
FLOAT32 10000000 10000000 246966x 5.089 us 345.98% 1.321 us 54.49% 7.568T 62.436 TB/s 8128.66%
FLOAT64 10000 10000 247163x 5.082 us 362.57% 1.319 us 52.32% 7.581G 123.190 GB/s 16.04%
FLOAT64 100000 100000 247047x 5.099 us 333.83% 1.317 us 52.22% 75.916G 1.234 TB/s 160.61%
FLOAT64 1000000 1000000 247002x 5.050 us 362.71% 1.319 us 52.95% 758.114G 12.319 TB/s 1603.88%
FLOAT64 10000000 10000000 246842x 5.103 us 367.78% 1.318 us 54.35% 7.588T 123.302 TB/s 16052.90%
BOOL8 10000 10000 56192x 12.672 us 66.69% 8.900 us 31.34% 1.124G 2.528 GB/s 0.33%
BOOL8 100000 100000 54608x 12.962 us 71.61% 9.158 us 36.73% 10.919G 24.567 GB/s 3.20%
BOOL8 1000000 1000000 38416x 16.701 us 52.25% 13.016 us 27.53% 76.831G 172.869 GB/s 22.51%
BOOL8 10000000 10000000 10048x 53.660 us 17.89% 49.800 us 12.16% 200.803G 451.806 GB/s 58.82%
TIMESTAMP_SECONDS 10000 10000 247149x 5.081 us 345.02% 1.317 us 54.13% 7.590G 123.344 GB/s 16.06%
TIMESTAMP_SECONDS 100000 100000 247242x 5.108 us 354.79% 1.320 us 52.39% 75.747G 1.231 TB/s 160.25%
TIMESTAMP_SECONDS 1000000 1000000 247343x 5.020 us 337.51% 1.321 us 51.63% 757.198G 12.304 TB/s 1601.94%
TIMESTAMP_SECONDS 10000000 10000000 246821x 5.083 us 355.62% 1.319 us 51.66% 7.581T 123.191 TB/s 16038.47%
TIMESTAMP_MILLISECONDS 10000 10000 247048x 5.084 us 350.33% 1.325 us 56.31% 7.548G 122.657 GB/s 15.97%
TIMESTAMP_MILLISECONDS 100000 100000 247106x 5.115 us 347.41% 1.320 us 55.73% 75.770G 1.231 TB/s 160.30%
TIMESTAMP_MILLISECONDS 1000000 1000000 247279x 5.040 us 340.21% 1.319 us 53.26% 757.897G 12.316 TB/s 1603.42%
TIMESTAMP_MILLISECONDS 10000000 10000000 246613x 5.100 us 385.60% 1.320 us 52.91% 7.575T 123.091 TB/s 16025.53%
TIMESTAMP_MICROSECONDS 10000 10000 247116x 5.087 us 338.60% 1.321 us 55.06% 7.572G 123.037 GB/s 16.02%
TIMESTAMP_MICROSECONDS 100000 100000 247149x 5.114 us 344.02% 1.320 us 53.72% 75.760G 1.231 TB/s 160.28%
TIMESTAMP_MICROSECONDS 1000000 1000000 247049x 5.056 us 357.31% 1.318 us 51.83% 758.837G 12.331 TB/s 1605.41%
TIMESTAMP_MICROSECONDS 10000000 10000000 246762x 5.096 us 339.52% 1.323 us 54.18% 7.556T 122.782 TB/s 15985.24%
TIMESTAMP_NANOSECONDS 10000 10000 246854x 5.100 us 351.21% 1.318 us 52.69% 7.586G 123.272 GB/s 16.05%
TIMESTAMP_NANOSECONDS 100000 100000 247222x 5.103 us 340.19% 1.322 us 54.80% 75.665G 1.230 TB/s 160.08%
TIMESTAMP_NANOSECONDS 1000000 1000000 247236x 5.031 us 345.68% 1.318 us 51.73% 758.893G 12.332 TB/s 1605.53%
TIMESTAMP_NANOSECONDS 10000000 10000000 247015x 5.073 us 338.08% 1.318 us 51.58% 7.587T 123.282 TB/s 16050.31%
DURATION_SECONDS 10000 10000 247078x 5.088 us 347.24% 1.319 us 53.32% 7.581G 123.184 GB/s 16.04%
DURATION_SECONDS 100000 100000 246937x 5.098 us 335.92% 1.324 us 55.92% 75.531G 1.227 TB/s 159.79%
DURATION_SECONDS 1000000 1000000 247259x 5.036 us 354.89% 1.322 us 55.16% 756.583G 12.294 TB/s 1600.64%
DURATION_SECONDS 10000000 10000000 246932x 5.080 us 338.82% 1.321 us 54.33% 7.570T 123.017 TB/s 16015.78%
DURATION_MILLISECONDS 10000 10000 246968x 5.087 us 341.71% 1.322 us 54.69% 7.565G 122.938 GB/s 16.01%
DURATION_MILLISECONDS 100000 100000 247064x 5.114 us 349.17% 1.320 us 54.61% 75.774G 1.231 TB/s 160.31%
DURATION_MILLISECONDS 1000000 1000000 247386x 5.046 us 346.93% 1.320 us 54.84% 757.740G 12.313 TB/s 1603.09%
DURATION_MILLISECONDS 10000000 10000000 246833x 5.087 us 340.08% 1.318 us 53.93% 7.587T 123.282 TB/s 16050.28%
DURATION_MICROSECONDS 10000 10000 247289x 5.085 us 352.21% 1.318 us 51.06% 7.587G 123.296 GB/s 16.05%
DURATION_MICROSECONDS 100000 100000 246804x 5.113 us 339.42% 1.317 us 51.35% 75.912G 1.234 TB/s 160.60%
DURATION_MICROSECONDS 1000000 1000000 246988x 5.039 us 341.89% 1.319 us 54.87% 758.414G 12.324 TB/s 1604.52%
DURATION_MICROSECONDS 10000000 10000000 246538x 5.089 us 366.05% 1.319 us 58.60% 7.580T 123.171 TB/s 16035.82%
DURATION_NANOSECONDS 10000 10000 246776x 5.091 us 340.70% 1.318 us 53.62% 7.587G 123.288 GB/s 16.05%
DURATION_NANOSECONDS 100000 100000 246864x 5.112 us 339.89% 1.319 us 53.01% 75.811G 1.232 TB/s 160.39%
DURATION_NANOSECONDS 1000000 1000000 246962x 5.032 us 338.37% 1.314 us 49.14% 761.090G 12.368 TB/s 1610.18%
DURATION_NANOSECONDS 10000000 10000000 246752x 5.087 us 345.50% 1.316 us 52.65% 7.596T 123.437 TB/s 16070.46%
STRING 10000 10000 246898x 5.079 us 341.33% 1.316 us 51.02% 7.601G 301.727 GB/s 39.28%
STRING 100000 100000 246596x 5.112 us 364.64% 1.323 us 59.79% 75.574G 2.993 TB/s 389.69%
STRING 1000000 1000000 246954x 5.031 us 338.05% 1.318 us 52.75% 758.773G 30.055 TB/s 3912.97%
STRING 10000000 10000000 246597x 5.090 us 340.87% 1.318 us 52.06% 7.586T 300.681 TB/s 39146.24%
LIST 10000 10000 22720x 26.030 us 33.77% 22.009 us 18.54% 454.351M 65.271 GB/s 8.50%
LIST 100000 100000 17408x 32.837 us 31.80% 28.748 us 17.64% 3.479G 501.203 GB/s 65.25%
LIST 1000000 1000000 5808x 90.215 us 11.93% 86.201 us 6.97% 11.601G 1.672 TB/s 217.71%
LIST 10000000 10000000 928x 660.223 us 0.92% 656.252 us 0.67% 15.238G 2.195 TB/s 285.83%
DECIMAL128 10000 10000 246917x 5.079 us 346.56% 1.316 us 53.60% 7.598G 245.047 GB/s 31.90%
DECIMAL128 100000 100000 246736x 5.115 us 379.70% 1.319 us 55.69% 75.799G 2.445 TB/s 318.26%
DECIMAL128 1000000 1000000 247021x 5.039 us 347.10% 1.317 us 51.70% 759.543G 24.495 TB/s 3189.09%
DECIMAL128 10000000 10000000 246521x 5.089 us 341.46% 1.322 us 59.05% 7.563T 243.912 TB/s 31755.42%
STRUCT 10000 10000 246696x 5.100 us 348.46% 1.321 us 69.80% 7.569G 424.610 GB/s 55.28%
STRUCT 100000 100000 246840x 5.112 us 356.58% 1.320 us 55.72% 75.740G 4.245 TB/s 552.63%
STRUCT 1000000 1000000 246923x 5.043 us 345.65% 1.322 us 54.51% 756.441G 42.401 TB/s 5520.28%
STRUCT 10000000 10000000 246429x 5.089 us 352.96% 1.325 us 65.44% 7.549T 423.330 TB/s 55114.23%

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@github-actions github-actions bot added libcudf Affects libcudf (C++/CUDA) code. CMake CMake build issue labels Oct 28, 2024
@lamarrr lamarrr added feature request New feature or request non-breaking Non-breaking change labels Oct 28, 2024
@lamarrr lamarrr changed the title Arrow Interop benchmarks Arrow Interop Benchmarks Oct 28, 2024
@lamarrr lamarrr changed the title Arrow Interop Benchmarks Added Arrow Interop Benchmarks Oct 28, 2024
@vyasr
Copy link
Contributor

vyasr commented Oct 31, 2024

Could you post some results? I'm expecting to_arrow_device to be almost free and to_arrow_host to be bounded by memcpy basically.

@lamarrr lamarrr marked this pull request as ready for review November 4, 2024 18:07
@lamarrr lamarrr requested a review from a team as a code owner November 4, 2024 18:07
@lamarrr lamarrr requested a review from ttnghia November 4, 2024 18:07
@ttnghia
Copy link
Contributor

ttnghia commented Nov 4, 2024

Can you post the benchmark comparison output by NVBENCH compare tool please? We cannot see anything from just the numbers like these.

@lamarrr
Copy link
Contributor Author

lamarrr commented Nov 4, 2024

Can you post the benchmark comparison output by NVBENCH compare tool please? We cannot see anything from just the numbers like these.

sure!

@lamarrr lamarrr marked this pull request as draft November 4, 2024 18:18
@lamarrr lamarrr marked this pull request as ready for review November 6, 2024 03:31
@GregoryKimball
Copy link
Contributor

I think this work is great to merge! Nice work

Some follow-ups I would consider:

  • why is LIST slower than flat types in from_arrow_device?
  • is it possible (and reasonable) to add support for DECIMAL32 and DECIMAL64?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vyasr vyasr Awaiting requested review from vyasr

@ttnghia ttnghia Awaiting requested review from ttnghia ttnghia is a code owner automatically assigned from rapidsai/cudf-cpp-codeowners

At least 1 approving review is required to merge this pull request.

Assignees

@lamarrr lamarrr

Labels
CMake CMake build issue feature request New feature or request libcudf Affects libcudf (C++/CUDA) code. non-breaking Non-breaking change
Projects
cuDF/Dask/Numba/UCX
Status: In Progress
libcudf
Status: Burndown
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@lamarrr @vyasr @ttnghia @GregoryKimball