[CB] Split token streaming and generation to different threads for all CB based pipelines #7245
| Job | Run time |
|---|---|
| 1m 20s | |
| 1m 26s | |
| 1m 25s | |
| 1m 25s | |
| 1m 40s | |
| 1m 23s | |
| 1m 42s | |
| 1m 41s | |
| 1m 31s | |
| 1m 22s | |
| 1m 23s | |
| 1m 22s | |
| 1m 17s | |
| 1m 38s | |
| 1m 20s | |
| 1m 22s | |
| 1m 44s | |
| 1m 43s | |
| 1m 40s | |
| 1m 31s | |
| 1m 28s | |
| 1s | |
| 31m 24s |