Simplified quick_abort_pct code and improved its docs

[vshailesh]

Authored-by: Shailesh Vashishth [email protected]

Instead of ignoring quick_abort_pct settings that would, together with
other conditions, abort a pending download of a 99-byte or smaller
response, Squid now honors quick_abort_pct for all response sizes. Most
Squids are not going to be affected by this change because default
quick_abort_min settings (16KB) prevent aborts of 99-byte responses even
before quick_abort_pct is checked.

Due to conversion from integer to floating point math, this change may
affect responses larger than 99 bytes as well, but these effects ought
to be limited to cases where the decision is based on a tiny difference
(e.g., receiving 1% more bytes would have triggered full download). In
most such cases, the decision could probably go either way due to
response header size fluctuations anyway.

Also updated quick_abort_pct documentation, primarily to clarify a
misleading statement: Squid did not and does not treat 16KB or smaller
responses specially in this context. The original statement was probably
based on quick_abort_min default setting of 16KB, but statement
phrasing and placement hid that connection.

[squid-prbot]

Can one of the admins verify this patch?

[rousskov]

Thank you for working on this ugly code! I support this change in principle (despite the fact that it replaces integer math with floating point math), but it needs a bit more work. Please see my change request for details.

[rousskov]

@vshailesh, please also add your contact information to CONTRIBUTORS file (to pass upcoming source-maintenance-tests CI check).

[rousskov]

Thank you for removing the no-longer-necessary code.

Please adjust PR description to briefly document the effect of that removal (e.g., I would expect a slight change in behavior for some setting of quick_abort_pct and possibly other quick_abort_... configuration options). When writing that comment, think as a sysadmin configuring Squid or reviewing their existing configuration (rather than as a Squid developer).

FYI: PR description formatting requirements are available, but most description formatting violations arise from exceeding the 72 character line length limit. Anubis will remove M-failed-description PR label a few minutes after you fix description formatting. If you cannot format PR description to pass Anubis check, do not worry about it -- we will do it for you.

I also expect that the source maintenance CI check will fail. Please use the diff provided in CI check output (follow the Details link) to adjust your CONTRIBUTORS modifications.

[vshailesh]

I have made changes to CONTRIBUTORS as suggested in the diff of CI check. Please retrigger the pipeline.

[rousskov]

I have made changes to CONTRIBUTORS as suggested in the diff of CI check. Please retrigger the pipeline.

I removed unwanted new line change (that change is visible in git diff 6c33c64..5239596 CONTRIBUTORS -- no need to run CI tests!). GitHub tests are running now.

Merge branch 'master' into SQUID-rewrite-math-expression

Just FYI: We do not need to keep PR branches in sync with master unless there are merge conflicts or other specific problems that require such synchronization. By default, ignore GitHub suggestions to merge master into your PR branch.

Alex: Please adjust PR description to briefly document the effect of that removal (e.g., I would expect a slight change in behavior for some setting of quick_abort_pct and possibly other quick_abort_... configuration options). When writing that comment, think as a sysadmin configuring Squid or reviewing their existing configuration (rather than as a Squid developer).

AFAICT, the above request is the only thing left before this PR can be cleared for merging. Thank you!

[rousskov]

Jenkins, it is OK to test this PR.

[rousskov]

@vshailesh, thank you for providing an example of changed Squid behavior. Your example focuses on quick_abort_pct calculations precision. You were right to disclose that change, but there is an arguably bigger change that the current example overlooks: Please add a brief statement describing whether/how the change affects quick_abort_pct behavior for responses that are smaller than 100 bytes in total message size. There is currently a contradiction between removed if statement and Squid documentation that this PR should clarify:

• On one hand, code removed by this PR clearly suggests that, prior to this PR, responses smaller than 100 bytes would continue to be fetched (as if quick_abort_pct was set to -1), but now quick_abort_pct is going to be honored for those responses.
• However, quick_abort_pct documentation in src/cf.data.pre continues to say "The cache by default continues downloading aborted requests which are ... less than 16 KB remaining". I do not see any relevant CheckQuickAbortIsReasonable() code, but perhaps that 16 KB check is elsewhere.

If Squid documentation is correct, then let's explicitly say (in the PR description) that the removed if statement had no effect on quick_abort_pct behavior (for such-and-such reasons). Otherwise, please remove that wrong statement from cf.data.pre and add a brief statement describing whether/how the change affects quick_abort_pct behavior for responses known to be smaller than 100 bytes. Needless to say, you should test to confirm these PR adjustments, whatever they are.

Once all the basic statements are in, I can help polish PR title/description, including its formatting, but I would prefer to rely on you to gather, describe, and validate the basic facts...

[rousskov]

This entry is just fine if you like it, but if you prefer to use some other email address, please note that you can adjust this entry accordingly. Our automation is "smart" enough to allow you to pick the email address you prefer...

[vshailesh]

I tried updating to another email, and I am not able to pick another email id. How do I do that?
The source maintenance check failed

[rousskov]

I am working on fixing this. Please wait.

[rousskov]

@vshailesh, almost done. The current PR branch passes local source maintenance tests, but fails the same tests on GitHub Actions. I know why, but I cannot fix that problem quickly.

For now, would you mind changing/setting your GitHub account email to the one you want to see in CONTRIBUTORS (hopefully, that is the same address that this PR branch has listed in CONTRIBUTORS now)? I hope (but cannot easily test) that such a GitHub account update will change the author of the merge commit that GitHub Actions test, resolving the last remaining problem.

We should be handling your use case better, but that will take time to fix... AFAIK, we have not seen it before.

[yadij]

Since this changes the behaviour of the Squid UI (config setting) this will also need a release notes mention for the quick_abort_pct directive.

[rousskov]

Amos: Since this changes the behaviour of the Squid UI (config setting) this will also need a release notes mention for the quick_abort_pct directive.

@vshailesh, I can help with updating release notes once PR description has the necessary disclosures as detailed in my earlier change request.

[vshailesh]

However, quick_abort_pct documentation in src/cf.data.pre continues to say "The cache by default continues downloading aborted requests which are ... less than 16 KB remaining". I do not see any relevant CheckQuickAbortIsReasonable() code, but perhaps that 16 KB check is elsewhere.

@rousskov I went through the code in detail and I did not find any check where we are checking the 16 KB (16384 bytes) statement for any of the quick_abort_* configs. The ConfigParser, store_client, StoreEntry and other places too, nowhere was a 16 KB check on these parameters. Although SQUID_UDP_SO_RCVBUF is set to 16384 bytes but I believe it is unrelated to these checks

Otherwise, please remove that wrong statement from cf.data.pre and add a brief statement describing whether/how the change affects quick_abort_pct behavior for responses known to be smaller than 100 bytes

We should remove the statement where it talks about 16 KB lower limit.

• Regarding behaviour for responses smaller than 100 bytes
  • You are correct, earlier we would have just downloaded the data if it was less than 100 bytes, to escape the division by zero problem. But now even 100 bytes will be checked against the quick_abort_pct parameter.

Needless to say, you should test to confirm these PR adjustments, whatever they are.

I would write some test to check this, and try to build and test on my local system with 100 bytes response, I settle on something and test this. If you meant something else by testing please tell me.

Since this changes the behaviour of the Squid UI (config setting) this will also need a release notes mention for the quick_abort_pct directive.

@yadij I would make these changes, once I am done with testing. I may need some help with this.

[rousskov]

Otherwise, please remove that wrong statement from cf.data.pre and add a brief statement describing whether/how the change affects quick_abort_pct behavior for responses known to be smaller than 100 bytes

We should remove the statement where it talks about 16 KB lower limit.

It sounds like we are on the same page. Good luck with these additional modifications! Please do not forget to keep PR description in sync with pushed PR branch code changes.

• You are correct, earlier we would have just downloaded the data if it was less than 100 bytes, to escape the division by zero problem. But now even 100 bytes will be checked against the quick_abort_pct parameter.

Please update PR description accordingly. This change in Squid behavior might be important in some esoteric use cases, and we have a duty to disclose it.

Needless to say, you should test to confirm these PR adjustments, whatever they are.

I would write some test to check this, and try to build and test on my local system with 100 bytes response, I settle on something and test this. If you meant something else by testing please tell me.

Your plan to test around 100-byte boundary sounds good to me! I would also test around 16KB boundary. Please do not forget to test both old/official and new/PR code to improve chances that your tests themselves are working as expected.

[vshailesh]

@rousskov
I was able to conclusively test and arrive at a result.

• I tested for both 100 bytes and 16 KB boundaries.
  Facts

1. 100 Bytes - No FPE boundary is real
   a. How to hit this boundary —
   i. set quick_abort_min 0 KB
   ii. set quick_abort_max 20000000000 KB (or some very large value)
   iii. set quick_abort_pct 99
   send data slow enough so that you have time to close the connection before transfer completes and you can see the No FPE exception in cache logs if debug_options directive is set to 90,3.

2. 16 KB boundary is not real i.e. we can abort a connection where we have less than 16 KB of total data (hdr_sz + content_length < 16 KB) and the CheckQuickAbortIsReasonable will go to the default abort condition. (The documentation states incorrectly that if the data is less than 16 KB, Squid will keep downloading even though client closes the connection)

Conclusion

1. We should update the documentation and remove the statements saying 16 KB boundary is real and also about people creating and aborting request thus tying up their file descriptor (because this would not happen, squid will close connection even if total data is less than 16 KB).

@yadij I have tried to update the release notes as well, please review them.
@rousskov If required I can give detailed explanation how I tested this, it took me a while to understand what to do.

[rousskov]

If required I can give detailed explanation how I tested this

Not required. Thank you for sharing the details you have already posted!

Please help me with M-failed-description

Done: The earlier description had a non-ASCII character in we’re.

I have also update PR title/description (after doing a bit more research on 16KB statement origins) and fixed a few small problems with documentation changes.

@vshailesh, please review my changes. If you disagree with any statements, please discuss. If you can improve anything, please do so.

[vshailesh]

@rousskov I was trying to fix Contributors file, that is why your approval got rejected, no other changes

[rousskov]

@vshailesh, almost done. The current PR branch passes local source maintenance tests, but fails the same tests on GitHub Actions. I know why, but I cannot fix that problem quickly.

For now, would you mind changing/setting your GitHub account email to the one you want to see in CONTRIBUTORS (hopefully, that is the same address that this PR branch has listed in CONTRIBUTORS now)? I hope (but cannot easily test) that such a GitHub account update will change the author of the merge commit that GitHub Actions test, resolving the last remaining problem.

We should be handling your use case better, but that will take time to fix... AFAIK, we have not seen it before.

[vshailesh]

@rousskov I have changed my Github settings

[vshailesh]

@rousskov Thank you for helping me out with this change.

[rousskov]

Thank you for helping me out with this change.

I am glad we are making progress. Now that PR tests are all green, I have cleared this PR for merging and will watch staging tests for failures.