RRUL Upload plot - why such low granularity?

[richb-hanover]

Here's an RRUL plot from my 7mbps/768kbps DSL circuit. The download has good granularity, while the upload plot seems only to have a half-dozen data points. This makes it seem faulty, or somehow different/disconnected from the other two plots.

I think I once saw a justification for the appearance of the upload plot, but I didn't understand it when I read it, and can't find it again to review. If this behavior is expected, let's document the reason.

The log and flent.gz files are at Rich_Test_gz_&_logs.zip Thanks.

[tohojo]

Rich Brown <[email protected]> writes:

Here's an RRUL plot from my 7mbps/768kbps DSL circuit. The download has good granularity, while the upload plot seems only to have a half-dozen data points. This makes it seem faulty, or somehow different/disconnected from the other two plots. I think I once saw a justification for the appearance of the upload plot, but I didn't understand it when I read it, and can't find it again to review.

Well, it's not really by design, but it's the best we can do with current tools, unfortunately. Basically, the problem is the way netperf determines its data output intervals in demo mode: It sends a certain number of bytes, and waits for that to complete before outputting anything. And, well, if the link is really slow, this takes a while, so we don't get a data point until it's done...

If this behavior is expected, let's document the reason.

Any suggestions as for where to put this so it's easy to find?

[richb-hanover]

answering the questions in the opposite order....

Any suggestions as for where to put this so it's easy to find?

The bufferbloat.net site has an RRUL Chart Explanation page. I have always felt the Flent site could have something like that as well, and this might be a good place to incorporate this information.

[richb-hanover]

Basically, the problem is the way netperf determines its data output intervals in demo mode:

Ahah! That was the explanation, but I wasn't (yet) smart enough to comprehend it. [To further test out my understanding... If netperf were to use a smaller "data output interval" when it's transmitting, it would create more frequent updates and the granularity would be higher...]

But I'm not sure that fully explains it unless netperf uses a (very) different data output interval for receiving... The download chart has hundreds of points, while the ratio between my download and upload is about 10:1 (7mbps vs 768kbps) so I'd expect the upload chart to have something like dozens of points...

Wait... maybe the chart really does have more points: each of the four plots above might have many inflection points but since fq_codel controls their rates so carefully, there's little change to their values, thus no obvious change to the charted values... And yes - turning off SQM (see image below) shows many more data points for those upload plots.

But are there enough points? In the download chart, I count around 30 samples for one plot in a 10-second period. So that's 3 per second. With an average data rate of 1.6mbps, it looks like 500 kbits/sample (a nice round number, arrived at with a SWAG - scientific wild ass guess.)

If the upload has the same data output interval, those upload streams (averaging 0.11 mbps) would need about 4.5 seconds to transmit that 500kbits. And if you squint at the plot below, and take into account the enormous SWAG in the previous paragraph, it looks OK.

All's right with the world. (But only after I remember to turn SQM back on...) Thanks.

[flent-users]

A little off topic but since pie is still a thing, and we have some new plot types...

…

On Thu, Sep 19, 2019 at 9:32 AM Rich Brown ***@***.***> wrote: Basically, the problem is the way netperf determines its data output intervals in demo mode: Ahah! That was the explanation, but I wasn't (yet) smart enough to comprehend it. [To further test out my understanding... If netperf were to use a smaller "data output interval" when it's transmitting, it would create more frequent updates and the granularity would be higher...] But I'm not sure that fully explains it unless netperf uses a (very) different data output interval for receiving... The download chart has hundreds of points, while the ratio between my download and upload is about 10:1 (7mbps vs 768kbps) so I'd expect the upload chart to have something like dozens of points... Wait... maybe the chart really does have more points: each of the four plots above might have many inflection points but since fq_codel controls their rates so carefully, there's little change to their values, thus no obvious change to the charted values... And yes - turning off SQM (see image below) shows many more data points for those upload plots. But are there enough points? In the download chart, I count around 30 samples for one plot in a 10-second period. So that's 3 per second. With an average data rate of 1.6mbps, it looks like 500 kbits/sample (a nice round number, arrived at with a SWAG - scientific wild ass guess.) If the upload has the same data output interval, those upload streams (averaging 0.11 mbps) would need about 4.5 seconds to transmit that 500kbits. And if you squint at the plot below, and take into account the enormous SWAG in the previous paragraph, it looks OK. All's right with the world. (But only after I remember to turn SQM back on...) Thanks. [image: image] <https://user-images.githubusercontent.com/1094930/65260618-a3f25380-dad4-11e9-880a-364522139d45.png> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#185?email_source=notifications&email_token=AHVNJP6N6Z7KYAGEUJL7A3LQKOSORA5CNFSM4IYMHHMKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7ECIOY#issuecomment-533210171>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHVNJP2VNCK5IP27GYKQNGTQKOSORANCNFSM4IYMHHMA> . _______________________________________________ Flent-users mailing list ***@***.*** http://flent.org/mailman/listinfo/flent-users_flent.org

-- Dave Täht CTO, TekLibre, LLC http://www.teklibre.com Tel: 1-831-205-9740

[tohojo]

Rich Brown <[email protected]> writes:

> Basically, the problem is the way netperf determines its data output intervals in demo mode: Ahah! That was the explanation, but I wasn't (yet) smart enough to comprehend it. [To further test out my understanding... If netperf were to use a smaller "data output interval" when it's transmitting, it would create more frequent updates and the granularity would be higher...] But I'm not sure that fully explains it unless netperf uses a (very) different data output interval for receiving... The download chart has hundreds of points, while the ratio between my download and upload is about 10:1 (7mbps vs 768kbps) so I'd expect the upload chart to have something like dozens of points...

This is because netperf only checks whether it should output a data point after it has sent a certain number of bytes. 16k in your case (it's in the series metdata). At 768kbps, 16k bytes takes ~2.6 seconds to send, so you'll get at most one data point for every such interval. Now that I'm looking at this again, it seems that you can actually set the send size. However, lowering it naturally impacts CPU usage. A quick test on my laptop indicates that lowering it to 512 bytes raises netperf's CPU usage from ~2% to ~20% for a single flow at half a gigabit. So yeah, we could probably lower it somewhat, but to what? We can't know ahead of time what connection speed we will be running on. If you want to try a quick test with a smaller size, this patch (to Flent) adds the option unconditionally; obviously not the right way to go about it, but it should allow you to test the impact at least: diff --git a/flent/runners.py b/flent/runners.py index a75223d..3d5e373 100644 --- a/flent/runners.py +++ b/flent/runners.py @@ -1049,6 +1049,7 @@ class NetperfDemoRunner(ProcessRunner): "{marking} -H {control_host} -p {control_port} " \ "-t {test} -l {length:d} {buffer} {format} " \ "{control_local_bind} {extra_args} -- " \ + "-m 512,512 " \ "{socket_timeout} {local_bind} -H {host} -k {output_vars} " \ "{cong_control} {extra_test_args}".format(**args)

[richb-hanover]

Sorry for the delay in responding... The higher granularity makes much better plots (see below). Using -m 2048,2048 I don't see a whole lot of load on my Mac 2.5 GHz Intel Core i7 at 7mbps/768kbps. Thanks.

[tohojo]

Rich Brown <[email protected]> writes:

Sorry for the delay in responding... The higher granularity makes much better plots (see below).

Great!

Using `-m 2048,2048` I don't see a whole lot of load on my Mac 2.5 GHz Intel Core i7 at 7mbps/768kbps. Thanks.

No, don't expect it would. The CPU usage thing will hit you at high rates (I was testing on a gigabit link). I'm not sure we can realistically pick an option that works well for both slow and fast links. So we may have to add a switch; and then the problem becomes what to use for defaults...

[flent-users]

Hi Toke,

On Sep 29, 2019, at 20:44, Toke Høiland-Jørgensen ***@***.***> wrote: Rich Brown ***@***.***> writes: > Sorry for the delay in responding... The higher granularity makes much > better plots (see below). Great! > Using `-m 2048,2048` I don't see a whole lot of load on my Mac 2.5 GHz > Intel Core i7 at 7mbps/768kbps. Thanks. No, don't expect it would. The CPU usage thing will hit you at high rates (I was testing on a gigabit link). I'm not sure we can realistically pick an option that works well for both slow and fast links. So we may have to add a switch; and then the problem becomes what to use for defaults...

How about make it not a "switch" but a numeric parameter, and default to the current default value by principle of least surprise? Best Regards Sebastian

…

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread. _______________________________________________ Flent-users mailing list ***@***.*** http://flent.org/mailman/listinfo/flent-users_flent.org

[tohojo]

flent-users <[email protected]> writes:

Hi Toke, > On Sep 29, 2019, at 20:44, Toke Høiland-Jørgensen ***@***.***> wrote: > > Rich Brown ***@***.***> writes: > > > Sorry for the delay in responding... The higher granularity makes much > > better plots (see below). > > Great! > > > Using `-m 2048,2048` I don't see a whole lot of load on my Mac 2.5 GHz > > Intel Core i7 at 7mbps/768kbps. Thanks. > > No, don't expect it would. The CPU usage thing will hit you at high > rates (I was testing on a gigabit link). > > I'm not sure we can realistically pick an option that works well for > both slow and fast links. So we may have to add a switch; and then the > problem becomes what to use for defaults... How about make it not a "switch" but a numeric parameter, and default to the current default value by principle of least surprise?

Well, by "switch" I just meant "new command line option". The obvious form of that would be, as you say, just the ability to set the netperf xfer size directly...

[flent-users]

Hi Toke,

On Sep 29, 2019, at 21:08, Toke Høiland-Jørgensen ***@***.***> wrote: flent-users ***@***.***> writes: > Hi Toke, > >> On Sep 29, 2019, at 20:44, Toke Høiland-Jørgensen ***@***.***> wrote: >> >> Rich Brown ***@***.***> writes: >> >> > Sorry for the delay in responding... The higher granularity makes much >> > better plots (see below). >> >> Great! >> >> > Using `-m 2048,2048` I don't see a whole lot of load on my Mac 2.5 GHz >> > Intel Core i7 at 7mbps/768kbps. Thanks. >> >> No, don't expect it would. The CPU usage thing will hit you at high >> rates (I was testing on a gigabit link). >> >> I'm not sure we can realistically pick an option that works well for >> both slow and fast links. So we may have to add a switch; and then the >> problem becomes what to use for defaults... > > How about make it not a "switch" but a numeric parameter, and > default to the current default value by principle of least > surprise? Well, by "switch" I just meant "new command line option".

Oops, sorry, I read this as a switch to toggle between say the default (which seems system dependent not petperf dependent, no?) and a value for low bandwidth links, say 1460...

The obvious form of that would be, as you say, just the ability to set the netperf xfer size directly...

I guess that would be the best compromise, if the user knows about a links asymmetry the send and receive buffers can be sized accordingly like 16K, 1600 for DOCSIS down-/upload... I also wonder whether flent should not try to also record and parse /proc/stat to detect CPU overload (so the user could be warned/informed to increase the xfer size if there is danger of being CPU bound)? Best Regards Sebastian

…

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread. _______________________________________________ Flent-users mailing list ***@***.*** http://flent.org/mailman/listinfo/flent-users_flent.org

[richb-hanover]

Well, by "switch" I just meant "new command line option". The obvious
form of that would be, as you say, just the ability to set the netperf
xfer size directly...

Yes. And let's remember why we're having this discussion - I asked why the charts for very slow links had so little granularity. RRUL has been this way since 2012, so it is not a "hotly-demanded" feature: we don't need to spend a whole lot of time on it.

Given that it's possible (when warranted) to tell netperf to increase its granularity, a Flent command line option would allow the person to request it if needed. My thoughts:

• The default for Flent should remain the same: defaults work fine for most people. The defaults also create datasets that are consistent with earlier test runs.
• The option should be named to suggest its purpose. I initially thought about "slow links" (because that kicked off this issue) but something along the lines of "increased granularity for slow links" might be better.
• The option should allow the person to specify the numbers directly (e.g. -m 2048,2048) to simplify implementation.
• The documentation should describe that this is the amount of data in bytes to be transferred between data samples; that using a smaller number increases granularity at the expense of higher CPU demand and increased data file size; provide guidance that the default is -m ???,??? (whatever is correct); and that -m 2048,2048 works fine for links under 1mbps.
• I suspect that the presence of two numbers indicates that one is for transmit, the other for receive. If so, the docs should specify which is which, and note that using the same value for both is reasonable.
• [NB] I was initially confused about the -m option - it doesn't seem to be described in the netperf documentation. Perhaps we could include a side note stating its purpose.