Vectorized CPU implementation for various unary functions for f32

[mert-kurttutan]

• Implemented vectorized implementation of math functions, sin, cos, tanh, exp, for f32 on CPU, with precision 1e-6
• Have some benchmarks showing 10-16% perf improvements on x86 machines, to be posted below
  I implemented in another crate since the testing of precision of these for various backend was easier

[mert-kurttutan]

To help reproducibility of benchmarks, I used whisper example as a benchmark.

cargo build --release --example "whisper" --features "symphonia"

# for powershell: Measure-Command { ./target/release/examples/whisper --input sample:gb1 | Out-Default }
# for single threaded linux: time taskset -c 0 ./target/release/examples/whisper --input sample:gb1
time ./target/release/examples/whisper --input sample:gb1

[mert-kurttutan]

Machine:

$ lscpu
Architecture:             x86_64
  CPU op-mode(s):         32-bit, 64-bit
  Address sizes:          39 bits physical, 48 bits virtual
  Byte Order:             Little Endian
CPU(s):                   12
  On-line CPU(s) list:    0-11
Vendor ID:                GenuineIntel
  Model name:             Intel(R) Core(TM) i5-10500H CPU @ 2.50GHz
    CPU family:           6
    Model:                165
    Thread(s) per core:   2
    Core(s) per socket:   6
    Socket(s):            1
    Stepping:             2
    CPU max MHz:          4500,0000
    CPU min MHz:          800,0000
    BogoMIPS:             4999.90
    Flags:                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge m
                          ca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 s
                          s ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc
                          art arch_perfmon pebs bts rep_good nopl xtopology nons
                          top_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor
                          ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid
                          sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer a
                          es xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpu
                          id_fault epb ssbd ibrs ibpb stibp ibrs_enhanced tpr_sh
                          adow flexpriority ept vpid ept_ad fsgsbase tsc_adjust
                          bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap c
                          lflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dthe
                          rm ida arat pln pts hwp hwp_notify hwp_act_window hwp_
                          epp vnmi pku ospke md_clear flush_l1d arch_capabilitie
                          s
Virtualization features:  
  Virtualization:         VT-x
Caches (sum of all):      
  L1d:                    192 KiB (6 instances)
  L1i:                    192 KiB (6 instances)
  L2:                     1,5 MiB (6 instances)
  L3:                     12 MiB (1 instance)

Single threaded:

# before change
time taskset -c 0 ./target/release/examples/whisper --input sample:gb1
Running on CPU, to run on GPU, build this example with `--features cuda`
pcm data loaded 3179927
loaded mel: [1, 80, 22500]
0.0s -- 30.0s:  My fellow Americans This day has brought terrible news and great sadness to our country And nine o'clock this morning Mission control in Houston lost contact with our space shuttle Columbia A short time later debris was seen falling from the skies above Texas The Columbia's lost There are no survivors
30.0s -- 60.0s:  Lord was a crew of seven Colonel Rick Husband Lieutenant Colonel Michael Anderson Commander Laurel Clark Captain David Brown Commander William McCool Doctor Kultna Shavla and Elan Ramon a Colonel in the Israeli Air Force These men and women assumed great risk in the service to all humanity in an age when space flight has
60.0s -- 90.0s:  It is easy to overlook the dangers of travel by rocket and the difficulties of navigating the fierce outer atmosphere of the earth. These astronauts knew the dangers and they faced them willingly, knowing they had a high and noble purpose in life. Because of their courage and daring and idealism, we will miss them all the more.
90.0s -- 120.0s:  All Americans today are thinking as well of the families of these men and women who have been given this sudden shock and grief You're not alone our entire nation agrees with you and those you loved will always have the respect and gratitude of this country The cause in which they died will continue the mankind is led into the darkness beyond our world
120.0s -- 150.0s:  by the inspiration of discovery and the longing to understand. Our journey into space will go on. In the skies today, we saw destruction and tragedy. As farther than we can see, there is comfort and hope. In the words of the prophet Isaiah, lift your eyes and look to the heavens who created all
150.0s -- 180.0s:  He who brings out the story hosts one by one and calls them each by name because of his great power and mighty strength not one of them is missing. The same creator who names the stars also knows the names of the seven souls we mourn today. The crew of the shuttle Columbia did not return safely to Earth.
180.0s -- 210.0s:  Yet we can pray that all are safely home. May God bless the grieving families. May God continue to bless America.
210.0s -- 225.0s:

real 0m29,112s
user 0m28,139s
sys 0m0,969s

# after change
time taskset -c 0 ./target/release/examples/whisper --input sample:gb1
Running on CPU, to run on GPU, build this example with `--features cuda`
pcm data loaded 3179927
loaded mel: [1, 80, 22500]
0.0s -- 30.0s:  My fellow Americans This day has brought terrible news and great sadness to our country And nine o'clock this morning Mission control in Houston lost contact with our space shuttle Columbia A short time later debris was seen falling from the skies above Texas The Columbia's lost There are no survivors
30.0s -- 60.0s:  Lord was a crew of seven Colonel Rick Husband Lieutenant Colonel Michael Anderson Commander Laurel Clark Captain David Brown Commander William McCool Doctor Kultna Shavla and Elan Ramon a Colonel in the Israeli Air Force These men and women assumed great risk in the service to all humanity in an age when space flight has
60.0s -- 90.0s:  It is easy to overlook the dangers of travel by rocket and the difficulties of navigating the fierce outer atmosphere of the earth. These astronauts knew the dangers and they faced them willingly, knowing they had a high and noble purpose in life. Because of their courage and daring and idealism, we will miss them all the more.
90.0s -- 120.0s:  All Americans today are thinking as well of the families of these men and women who have been given this sudden shock and grief You're not alone our entire nation agrees with you and those you loved will always have the respect and gratitude of this country The cause in which they died will continue the mankind is led into the darkness beyond our world
120.0s -- 150.0s:  by the inspiration of discovery and the longing to understand. Our journey into space will go on. In the skies today, we saw destruction and tragedy. As farther than we can see, there is comfort and hope. In the words of the prophet Isaiah, lift your eyes and look to the heavens who created all
150.0s -- 180.0s:  He who brings out the story hosts one by one and calls them each by name because of his great power and mighty strength not one of them is missing. The same creator who names the stars also knows the names of the seven souls we mourn today. The crew of the shuttle Columbia did not return safely to Earth.
180.0s -- 210.0s:  Yet we can pray that all are safely home. May God bless the grieving families. May God continue to bless America.
210.0s -- 225.0s:

real 0m25,827s
user 0m24,785s
sys 0m1,034s

Multi threaded:

# before change
time ./target/release/examples/whisper --input sample:gb1
Running on CPU, to run on GPU, build this example with `--features cuda`
pcm data loaded 3179927
loaded mel: [1, 80, 22500]
0.0s -- 30.0s:  My fellow Americans This day has brought terrible news and great sadness to our country And nine o'clock this morning Mission control in Houston lost contact with our space shuttle Columbia A short time later debris was seen falling from the skies above Texas The Columbia's lost There are no survivors
30.0s -- 60.0s:  Lord was a crew of seven Colonel Rick Husband Lieutenant Colonel Michael Anderson Commander Laurel Clark Captain David Brown Commander William McCool Doctor Kultna Shavla and Elan Ramon a Colonel in the Israeli Air Force These men and women assumed great risk in the service to all humanity in an age when space flight has
60.0s -- 90.0s:  It is easy to overlook the dangers of travel by rocket and the difficulties of navigating the fierce outer atmosphere of the earth. These astronauts knew the dangers and they faced them willingly, knowing they had a high and noble purpose in life. Because of their courage and daring and idealism, we will miss them all the more.
90.0s -- 120.0s:  All Americans today are thinking as well of the families of these men and women who have been given this sudden shock and grief You're not alone our entire nation agrees with you and those you loved will always have the respect and gratitude of this country The cause in which they died will continue the mankind is led into the darkness beyond our world
120.0s -- 150.0s:  by the inspiration of discovery and the longing to understand. Our journey into space will go on. In the skies today, we saw destruction and tragedy. As farther than we can see, there is comfort and hope. In the words of the prophet Isaiah, lift your eyes and look to the heavens who created all
150.0s -- 180.0s:  He who brings out the story hosts one by one and calls them each by name because of his great power and mighty strength not one of them is missing. The same creator who names the stars also knows the names of the seven souls we mourn today. The crew of the shuttle Columbia did not return safely to Earth.
180.0s -- 210.0s:  Yet we can pray that all are safely home. May God bless the grieving families. May God continue to bless America.
210.0s -- 225.0s:

real 0m15,158s
user 0m50,629s
sys 0m8,159s

# after change
time ./target/release/examples/whisper --input sample:gb1
Running on CPU, to run on GPU, build this example with `--features cuda`
pcm data loaded 3179927
loaded mel: [1, 80, 22500]
0.0s -- 30.0s:  My fellow Americans This day has brought terrible news and great sadness to our country And nine o'clock this morning Mission control in Houston lost contact with our space shuttle Columbia A short time later debris was seen falling from the skies above Texas The Columbia's lost There are no survivors
30.0s -- 60.0s:  Lord was a crew of seven Colonel Rick Husband Lieutenant Colonel Michael Anderson Commander Laurel Clark Captain David Brown Commander William McCool Doctor Kultna Shavla and Elan Ramon a Colonel in the Israeli Air Force These men and women assumed great risk in the service to all humanity in an age when space flight has
60.0s -- 90.0s:  It is easy to overlook the dangers of travel by rocket and the difficulties of navigating the fierce outer atmosphere of the earth. These astronauts knew the dangers and they faced them willingly, knowing they had a high and noble purpose in life. Because of their courage and daring and idealism, we will miss them all the more.
90.0s -- 120.0s:  All Americans today are thinking as well of the families of these men and women who have been given this sudden shock and grief You're not alone our entire nation agrees with you and those you loved will always have the respect and gratitude of this country The cause in which they died will continue the mankind is led into the darkness beyond our world
120.0s -- 150.0s:  by the inspiration of discovery and the longing to understand. Our journey into space will go on. In the skies today, we saw destruction and tragedy. As farther than we can see, there is comfort and hope. In the words of the prophet Isaiah, lift your eyes and look to the heavens who created all
150.0s -- 180.0s:  He who brings out the story hosts one by one and calls them each by name because of his great power and mighty strength not one of them is missing. The same creator who names the stars also knows the names of the seven souls we mourn today. The crew of the shuttle Columbia did not return safely to Earth.
180.0s -- 210.0s:  Yet we can pray that all are safely home. May God bless the grieving families. May God continue to bless America.
210.0s -- 225.0s:

real 0m11,747s
user 0m47,459s
sys 0m8,434s

[mert-kurttutan]

Some arm machine:

lscpu
Architecture:                         aarch64
CPU op-mode(s):                       64-bit
Byte Order:                           Little Endian
CPU(s):                               4
On-line CPU(s) list:                  0-3
Thread(s) per core:                   1
Core(s) per socket:                   4
Socket(s):                            1
NUMA node(s):                         1
Vendor ID:                            ARM
Model:                                1
Stepping:                             r0p1
BogoMIPS:                             2000.00
L1d cache:                            256 KiB
L1i cache:                            256 KiB
L2 cache:                             8 MiB
L3 cache:                             80 MiB
NUMA node0 CPU(s):                    0-3
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Not affected
Vulnerability Spec rstack overflow:   Not affected
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:             Mitigation; __user pointer sanitization
Vulnerability Spectre v2:             Not affected
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected
Flags:                                fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asim
                                      drdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit u
                                      scat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm sves
                                      ha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti

Single threaded:

# before change
time taskset -c 0 ./target/release/examples/whisper --input sample:gb1
Running on CPU, to run on GPU, build this example with `--features cuda`
pcm data loaded 3179927
loaded mel: [1, 80, 22500]
0.0s -- 30.0s:  My fellow Americans This day has brought terrible news and great sadness to our country And nine o'clock this morning Mission control in Houston lost contact with our space shuttle Columbia A short time later debris was seen falling from the skies above Texas The Columbia's lost There are no survivors
30.0s -- 60.0s:  Lord was a crew of seven Colonel Rick Husband Lieutenant Colonel Michael Anderson Commander Laurel Clark Captain David Brown Commander William McCool Doctor Kultna Shavla and Elan Ramon a Colonel in the Israeli Air Force These men and women assumed great risk in the service to all humanity in an age when space flight has
60.0s -- 90.0s:  It is easy to overlook the dangers of travel by rocket and the difficulties of navigating the fierce outer atmosphere of the earth. These astronauts knew the dangers and they faced them willingly, knowing they had a high and noble purpose in life. Because of their courage and daring and idealism, we will miss them all the more.
90.0s -- 120.0s:  All Americans today are thinking as well of the families of these men and women who have been given this sudden shock and grief You're not alone our entire nation agrees with you and those you loved will always have the respect and gratitude of this country The cause in which they died will continue the mankind is led into the darkness beyond our world
120.0s -- 150.0s:  by the inspiration of discovery and the longing to understand. Our journey into space will go on. In the skies today, we saw destruction and tragedy. As farther than we can see, there is comfort and hope. In the words of the prophet Isaiah, lift your eyes and look to the heavens who created all
150.0s -- 180.0s:  He who brings out the story hosts one by one and calls them each by name because of his great power and mighty strength not one of them is missing. The same creator who names the stars also knows the names of the seven souls we mourn today. The crew of the shuttle Columbia did not return safely to Earth.
180.0s -- 210.0s:  Yet we can pray that all are safely home. May God bless the grieving families. May God continue to bless America.
210.0s -- 225.0s: 

real    0m23.715s
user    0m22.826s
sys     0m0.888s

# after change
time taskset -c 0 ./target/release/examples/whisper --input sample:gb1
Running on CPU, to run on GPU, build this example with `--features cuda`
pcm data loaded 3179927
loaded mel: [1, 80, 22500]
0.0s -- 30.0s:  My fellow Americans This day has brought terrible news and great sadness to our country And nine o'clock this morning Mission control in Houston lost contact with our space shuttle Columbia A short time later debris was seen falling from the skies above Texas The Columbia's lost There are no survivors
30.0s -- 60.0s:  Lord was a crew of seven Colonel Rick Husband Lieutenant Colonel Michael Anderson Commander Laurel Clark Captain David Brown Commander William McCool Doctor Kultna Shavla and Elan Ramon a Colonel in the Israeli Air Force These men and women assumed great risk in the service to all humanity in an age when space flight has
60.0s -- 90.0s:  It is easy to overlook the dangers of travel by rocket and the difficulties of navigating the fierce outer atmosphere of the earth. These astronauts knew the dangers and they faced them willingly, knowing they had a high and noble purpose in life. Because of their courage and daring and idealism, we will miss them all the more.
90.0s -- 120.0s:  All Americans today are thinking as well of the families of these men and women who have been given this sudden shock and grief You're not alone our entire nation agrees with you and those you loved will always have the respect and gratitude of this country The cause in which they died will continue the mankind is led into the darkness beyond our world
120.0s -- 150.0s:  by the inspiration of discovery and the longing to understand. Our journey into space will go on. In the skies today, we saw destruction and tragedy. As farther than we can see, there is comfort and hope. In the words of the prophet Isaiah, lift your eyes and look to the heavens who created all
150.0s -- 180.0s:  He who brings out the story hosts one by one and calls them each by name because of his great power and mighty strength not one of them is missing. The same creator who names the stars also knows the names of the seven souls we mourn today. The crew of the shuttle Columbia did not return safely to Earth.
180.0s -- 210.0s:  Yet we can pray that all are safely home. May God bless the grieving families. May God continue to bless America.
210.0s -- 225.0s: 

real    0m20.797s
user    0m19.863s
sys     0m0.932s

Multi threaded:

# before change
time ./target/release/examples/whisper --input sample:gb1
Running on CPU, to run on GPU, build this example with `--features cuda`
pcm data loaded 3179927
loaded mel: [1, 80, 22500]
0.0s -- 30.0s:  My fellow Americans This day has brought terrible news and great sadness to our country And nine o'clock this morning Mission control in Houston lost contact with our space shuttle Columbia A short time later debris was seen falling from the skies above Texas The Columbia's lost There are no survivors
30.0s -- 60.0s:  Lord was a crew of seven Colonel Rick Husband Lieutenant Colonel Michael Anderson Commander Laurel Clark Captain David Brown Commander William McCool Doctor Kultna Shavla and Elan Ramon a Colonel in the Israeli Air Force These men and women assumed great risk in the service to all humanity in an age when space flight has
60.0s -- 90.0s:  It is easy to overlook the dangers of travel by rocket and the difficulties of navigating the fierce outer atmosphere of the earth. These astronauts knew the dangers and they faced them willingly, knowing they had a high and noble purpose in life. Because of their courage and daring and idealism, we will miss them all the more.
90.0s -- 120.0s:  All Americans today are thinking as well of the families of these men and women who have been given this sudden shock and grief You're not alone our entire nation agrees with you and those you loved will always have the respect and gratitude of this country The cause in which they died will continue the mankind is led into the darkness beyond our world
120.0s -- 150.0s:  by the inspiration of discovery and the longing to understand. Our journey into space will go on. In the skies today, we saw destruction and tragedy. As farther than we can see, there is comfort and hope. In the words of the prophet Isaiah, lift your eyes and look to the heavens who created all
150.0s -- 180.0s:  He who brings out the story hosts one by one and calls them each by name because of his great power and mighty strength not one of them is missing. The same creator who names the stars also knows the names of the seven souls we mourn today. The crew of the shuttle Columbia did not return safely to Earth.
180.0s -- 210.0s:  Yet we can pray that all are safely home. May God bless the grieving families. May God continue to bless America.
210.0s -- 225.0s: 

real    0m14.897s
user    0m26.563s
sys     0m5.795s

# after change
time ./target/release/examples/whisper --input sample:gb1
Running on CPU, to run on GPU, build this example with `--features cuda`
pcm data loaded 3179927
loaded mel: [1, 80, 22500]
0.0s -- 30.0s:  My fellow Americans This day has brought terrible news and great sadness to our country And nine o'clock this morning Mission control in Houston lost contact with our space shuttle Columbia A short time later debris was seen falling from the skies above Texas The Columbia's lost There are no survivors
30.0s -- 60.0s:  Lord was a crew of seven Colonel Rick Husband Lieutenant Colonel Michael Anderson Commander Laurel Clark Captain David Brown Commander William McCool Doctor Kultna Shavla and Elan Ramon a Colonel in the Israeli Air Force These men and women assumed great risk in the service to all humanity in an age when space flight has
60.0s -- 90.0s:  It is easy to overlook the dangers of travel by rocket and the difficulties of navigating the fierce outer atmosphere of the earth. These astronauts knew the dangers and they faced them willingly, knowing they had a high and noble purpose in life. Because of their courage and daring and idealism, we will miss them all the more.
90.0s -- 120.0s:  All Americans today are thinking as well of the families of these men and women who have been given this sudden shock and grief You're not alone our entire nation agrees with you and those you loved will always have the respect and gratitude of this country The cause in which they died will continue the mankind is led into the darkness beyond our world
120.0s -- 150.0s:  by the inspiration of discovery and the longing to understand. Our journey into space will go on. In the skies today, we saw destruction and tragedy. As farther than we can see, there is comfort and hope. In the words of the prophet Isaiah, lift your eyes and look to the heavens who created all
150.0s -- 180.0s:  He who brings out the story hosts one by one and calls them each by name because of his great power and mighty strength not one of them is missing. The same creator who names the stars also knows the names of the seven souls we mourn today. The crew of the shuttle Columbia did not return safely to Earth.
180.0s -- 210.0s:  Yet we can pray that all are safely home. May God bless the grieving families. May God continue to bless America.
210.0s -- 225.0s: 

real    0m11.153s
user    0m22.584s
sys     0m4.635s

[mert-kurttutan]

I could add some vectorized implementation for binary function. But, these were not as significant as vectorization of more complex math functions (e.g. exp, tanh) since auto vectorization with sse2 features were good enough.

[mert-kurttutan]

@LaurentMazare you can let me know if there is any additional thing needed.

[mert-kurttutan]

Anybody out there for a feedback/review?

[EricLBuehler]

Hey @mert-kurttutan! These changes seem interesting. I have yet to do some benchmarks, but I spotted something that seems incorrect.

[mert-kurttutan]

Hey @mert-kurttutan! These changes seem interesting. I have yet to do some benchmarks, but I spotted something that seems incorrect.

Is there a official end-to-end benchmark?