Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: More migration tests and related fixes #2170

Merged
merged 27 commits into from
Nov 14, 2024
Merged

Conversation

larseggert
Copy link
Collaborator

@larseggert larseggert commented Oct 11, 2024

This expands the migration test quite a bit. It now tests all the way until retirement of the original path, for both port-only (rebinding_port) and address-and-port (rebinding_address_and_port) changes, with and without zero-len client CIDs.

This now includes #2182 and #2176.

This expands the migration test quite a bit. It now tests alll the way
until retirement of the original path, for both port-only
(`rebinding_port`) and address-and-port (`rebinding_address_and_port`)
changes.

`rebinding_address_and_port` is succeeding, but `rebinding_port` is
currently failing. That's because we treat it differently for some
reason. If we replace `Paths::find_path_with_rebinding` with
`Paths::find_path`, i.e., do proper path validation when only the port
changes, the test succeeds.

Leaving this out in case I'm missing something about the intent of the
difference.
@larseggert
Copy link
Collaborator Author

In a related QNS test, we have the problem that the client does not issue a new connection ID when the server retires the path, and then we time out because the client wont accept any more packets from the server. I was hoping to reproduce this with this test, but it's behaving differently. Sigh.

Copy link

github-actions bot commented Oct 11, 2024

Failed Interop Tests

QUIC Interop Runner, client vs. server

neqo-latest as client

neqo-latest as server

All results

Succeeded Interop Tests

QUIC Interop Runner, client vs. server

neqo-latest as client

neqo-latest as server

Unsupported Interop Tests

QUIC Interop Runner, client vs. server

neqo-latest as client

neqo-latest as server

Copy link

codecov bot commented Oct 18, 2024

Codecov Report

Attention: Patch coverage is 94.59459% with 2 lines in your changes missing coverage. Please review.

Project coverage is 95.36%. Comparing base (978aa4e) to head (0fe2219).
Report is 3 commits behind head on main.

Files with missing lines Patch % Lines
neqo-transport/src/cid.rs 93.75% 1 Missing ⚠️
neqo-transport/src/path.rs 94.11% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2170      +/-   ##
==========================================
- Coverage   95.37%   95.36%   -0.01%     
==========================================
  Files         112      112              
  Lines       36449    36427      -22     
==========================================
- Hits        34762    34738      -24     
- Misses       1687     1689       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@larseggert larseggert changed the title test: More migration testing fix: More migration tests and related fixes Oct 18, 2024
@larseggert larseggert marked this pull request as ready for review October 18, 2024 14:49
Copy link
Collaborator

@mxinden mxinden left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for expanding the test matrix!

neqo-transport/src/cid.rs Show resolved Hide resolved
neqo-transport/src/cid.rs Outdated Show resolved Hide resolved
neqo-transport/src/connection/tests/migration.rs Outdated Show resolved Hide resolved
neqo-transport/src/connection/tests/migration.rs Outdated Show resolved Hide resolved
neqo-transport/src/connection/mod.rs Show resolved Hide resolved
Copy link

github-actions bot commented Oct 21, 2024

Benchmark results

Performance differences relative to 978aa4e.

coalesce_acked_from_zero 1+1 entries: 💔 Performance has regressed.
       time:   [105.60 ns 105.99 ns 106.38 ns]
       change: [+6.4418% +6.9710% +7.4688%] (p = 0.00 < 0.05)

Found 16 outliers among 100 measurements (16.00%)
11 (11.00%) high mild
5 (5.00%) high severe

coalesce_acked_from_zero 3+1 entries: 💔 Performance has regressed.
       time:   [121.28 ns 121.54 ns 121.85 ns]
       change: [+3.2560% +3.6974% +4.1287%] (p = 0.00 < 0.05)

Found 17 outliers among 100 measurements (17.00%)
2 (2.00%) low severe
3 (3.00%) low mild
3 (3.00%) high mild
9 (9.00%) high severe

coalesce_acked_from_zero 10+1 entries: 💔 Performance has regressed.
       time:   [120.97 ns 121.34 ns 121.83 ns]
       change: [+2.9226% +3.9609% +5.1498%] (p = 0.00 < 0.05)

Found 23 outliers among 100 measurements (23.00%)
11 (11.00%) low severe
1 (1.00%) low mild
3 (3.00%) high mild
8 (8.00%) high severe

coalesce_acked_from_zero 1000+1 entries: 💔 Performance has regressed.
       time:   [100.34 ns 100.58 ns 100.86 ns]
       change: [+1.7375% +2.5903% +3.4053%] (p = 0.00 < 0.05)

Found 12 outliers among 100 measurements (12.00%)
6 (6.00%) high mild
6 (6.00%) high severe

RxStreamOrderer::inbound_frame(): Change within noise threshold.
       time:   [112.01 ms 112.06 ms 112.12 ms]
       change: [-0.5045% -0.3071% -0.1760%] (p = 0.00 < 0.05)

Found 16 outliers among 100 measurements (16.00%)
8 (8.00%) low mild
8 (8.00%) high mild

transfer/pacing-false/varying-seeds: No change in performance detected.
       time:   [26.330 ms 27.320 ms 28.314 ms]
       change: [-6.4928% -1.2695% +4.2777%] (p = 0.65 > 0.05)

Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high mild

transfer/pacing-true/varying-seeds: No change in performance detected.
       time:   [33.805 ms 35.468 ms 37.131 ms]
       change: [-11.482% -5.5438% +0.0689%] (p = 0.08 > 0.05)
transfer/pacing-false/same-seed: No change in performance detected.
       time:   [25.137 ms 25.895 ms 26.647 ms]
       change: [-5.3764% -1.1632% +2.9220%] (p = 0.59 > 0.05)
transfer/pacing-true/same-seed: No change in performance detected.
       time:   [40.075 ms 42.002 ms 43.979 ms]
       change: [-9.1792% -2.7744% +4.0764%] (p = 0.44 > 0.05)
1-conn/1-100mb-resp/mtu-1500 (aka. Download)/client: No change in performance detected.
       time:   [939.98 ms 948.91 ms 958.02 ms]
       thrpt:  [104.38 MiB/s 105.38 MiB/s 106.39 MiB/s]
change:
       time:   [-1.3558% -0.1261% +1.0940%] (p = 0.85 > 0.05)
       thrpt:  [-1.0821% +0.1263% +1.3745%]
1-conn/10_000-parallel-1b-resp/mtu-1500 (aka. RPS)/client: 💔 Performance has regressed.
       time:   [326.60 ms 329.51 ms 332.51 ms]
       thrpt:  [30.074 Kelem/s 30.348 Kelem/s 30.619 Kelem/s]
change:
       time:   [+2.0918% +3.4934% +4.8172%] (p = 0.00 < 0.05)
       thrpt:  [-4.5958% -3.3755% -2.0489%]
1-conn/1-1b-resp/mtu-1500 (aka. HPS)/client: No change in performance detected.
       time:   [33.511 ms 33.700 ms 33.905 ms]
       thrpt:  [29.494  elem/s 29.674  elem/s 29.841  elem/s]
change:
       time:   [-0.9949% -0.1428% +0.7046%] (p = 0.75 > 0.05)
       thrpt:  [-0.6997% +0.1430% +1.0049%]

Found 4 outliers among 100 measurements (4.00%)
2 (2.00%) high mild
2 (2.00%) high severe

1-conn/1-100mb-resp/mtu-1500 (aka. Upload)/client: 💔 Performance has regressed.
       time:   [1.6682 s 1.6906 s 1.7141 s]
       thrpt:  [58.340 MiB/s 59.151 MiB/s 59.945 MiB/s]
change:
       time:   [+1.8420% +3.7645% +5.7478%] (p = 0.00 < 0.05)
       thrpt:  [-5.4354% -3.6279% -1.8087%]

Found 3 outliers among 100 measurements (3.00%)
3 (3.00%) high mild

1-conn/1-100mb-resp/mtu-65536 (aka. Download)/client: No change in performance detected.
       time:   [112.25 ms 112.50 ms 112.74 ms]
       thrpt:  [887.01 MiB/s 888.91 MiB/s 890.87 MiB/s]
change:
       time:   [-0.4400% -0.1222% +0.2081%] (p = 0.47 > 0.05)
       thrpt:  [-0.2077% +0.1223% +0.4419%]

Found 2 outliers among 100 measurements (2.00%)
2 (2.00%) low mild

1-conn/10_000-parallel-1b-resp/mtu-65536 (aka. RPS)/client: No change in performance detected.
       time:   [316.49 ms 319.67 ms 322.76 ms]
       thrpt:  [30.982 Kelem/s 31.282 Kelem/s 31.596 Kelem/s]
change:
       time:   [-2.0843% -0.6487% +0.7861%] (p = 0.38 > 0.05)
       thrpt:  [-0.7800% +0.6529% +2.1286%]

Found 2 outliers among 100 measurements (2.00%)
2 (2.00%) low mild

1-conn/1-1b-resp/mtu-65536 (aka. HPS)/client: 💔 Performance has regressed.
       time:   [34.350 ms 34.533 ms 34.735 ms]
       thrpt:  [28.790  elem/s 28.958  elem/s 29.112  elem/s]
change:
       time:   [+1.2829% +2.1211% +3.0221%] (p = 0.00 < 0.05)
       thrpt:  [-2.9335% -2.0771% -1.2666%]

Found 6 outliers among 100 measurements (6.00%)
1 (1.00%) low mild
1 (1.00%) high mild
4 (4.00%) high severe

1-conn/1-100mb-resp/mtu-65536 (aka. Upload)/client: No change in performance detected.
       time:   [279.66 ms 359.17 ms 453.86 ms]
       thrpt:  [220.33 MiB/s 278.42 MiB/s 357.58 MiB/s]
change:
       time:   [-13.622% +18.020% +62.614%] (p = 0.33 > 0.05)
       thrpt:  [-38.505% -15.269% +15.770%]

Found 9 outliers among 100 measurements (9.00%)
2 (2.00%) high mild
7 (7.00%) high severe

Client/server transfer results

Transfer of 33554432 bytes over loopback.

Client Server CC Pacing MTU Mean [ms] Min [ms] Max [ms] Relative
msquic msquic 1504 237.1 ± 98.0 102.3 332.2 1.00
neqo msquic reno on 1504 248.5 ± 54.4 209.6 404.0 1.00
neqo msquic reno 1504 214.1 ± 16.0 194.8 251.4 1.00
neqo msquic cubic on 1504 212.1 ± 12.6 195.0 233.7 1.00
neqo msquic cubic 1504 280.7 ± 85.4 202.5 416.6 1.00
msquic neqo reno on 1504 709.5 ± 10.9 694.3 729.5 1.00
msquic neqo reno 1504 734.9 ± 73.3 683.6 912.6 1.00
msquic neqo cubic on 1504 705.2 ± 12.1 693.6 728.9 1.00
msquic neqo cubic 1504 719.5 ± 33.7 666.7 797.8 1.00
neqo neqo reno on 1504 442.9 ± 17.7 420.6 477.9 1.00
neqo neqo reno 1504 458.4 ± 54.6 422.8 610.4 1.00
neqo neqo cubic on 1504 476.6 ± 76.2 418.3 620.3 1.00
neqo neqo cubic 1504 421.7 ± 17.5 400.2 454.5 1.00
msquic msquic 65536 168.7 ± 90.8 101.9 348.0 1.00
neqo msquic reno on 65536 288.3 ± 94.6 208.8 431.1 1.00
neqo msquic reno 65536 211.6 ± 12.5 199.7 235.1 1.00
neqo msquic cubic on 65536 211.3 ± 10.2 197.7 233.1 1.00
neqo msquic cubic 65536 257.4 ± 97.6 198.3 441.1 1.00
msquic neqo reno on 65536 158.4 ± 104.1 82.7 437.3 1.00
msquic neqo reno 65536 139.4 ± 85.5 82.8 331.0 1.00
msquic neqo cubic on 65536 129.1 ± 87.6 82.0 355.8 1.00
msquic neqo cubic 65536 132.6 ± 79.6 82.3 328.7 1.00
neqo neqo reno on 65536 182.2 ± 90.9 118.3 428.2 1.00
neqo neqo reno 65536 159.3 ± 66.1 126.1 406.8 1.00
neqo neqo cubic on 65536 180.8 ± 103.0 123.0 438.0 1.00
neqo neqo cubic 65536 139.5 ± 17.3 119.2 175.4 1.00

⬇️ Download logs

@larseggert larseggert requested a review from mxinden October 24, 2024 11:23
Copy link
Collaborator

@mxinden mxinden left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple of clarifying questions. Sorry in case I am missing something obvious. None of them are blocking.

Other than the discussion in #2176 (review), this looks good to me.

neqo-transport/src/cid.rs Show resolved Hide resolved
neqo-transport/src/path.rs Outdated Show resolved Hide resolved
neqo-transport/src/path.rs Outdated Show resolved Hide resolved
neqo-transport/src/path.rs Outdated Show resolved Hide resolved
neqo-transport/src/cid.rs Show resolved Hide resolved
@larseggert larseggert requested a review from mxinden October 25, 2024 10:46
neqo-transport/src/cid.rs Show resolved Hide resolved
@larseggert larseggert enabled auto-merge November 14, 2024 14:28
@larseggert larseggert disabled auto-merge November 14, 2024 14:28
@larseggert larseggert merged commit 68e19fc into main Nov 14, 2024
62 of 64 checks passed
@larseggert larseggert deleted the test-more-migration branch November 14, 2024 14:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants