-
Notifications
You must be signed in to change notification settings - Fork 291
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Documentation on sub-models #95
Comments
I agree that the example is a little confusing. The trend is going to be more visible with larger models. E.g. add a few more parameters to it to see the real difference. I also agree that ***@***.***” is a simpler and cleater illustration of the principle with a model as small as this.
From: Dan Tripp ***@***.***>
Date: Friday, December 2, 2022 at 8:27 AM
To: microsoft/pict ***@***.***>
Cc: Subscribed ***@***.***>
Subject: [microsoft/pict] Documentation on sub-models (Issue #95)
There's something that I don't understand in the documentation in pict.md<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2Fpict%2Fblob%2F18945b31394573fd01ce54b24fbf878968c7681f%2Fdoc%2Fpict.md&data=05%7C01%7Cjacekcz%40microsoft.com%7C0c93a70f7b894417cb5308dad4822483%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638055952673283035%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=HWM5zpbWMY0MHoVLDtYgCO3IbUORHjMUNgLo7r8XjF0%3D&reserved=0>, where it says:
"Placing all hardware parameters into one sub-model produces fewer distinct hardware configurations and potentially lowers the cost of testing."
But the example shown (with { PLATFORM, CPUS, RAM, HDD } @ 3) seems to produce more distinct hardware configurations. Not fewer.
By my count:
{ PLATFORM, CPUS, RAM, HDD } @ 3 produces 28 hardware configurations (across 336 generated tests.)
{ PLATFORM, CPUS, RAM, HDD } @ 2 produces 10 hardware configurations (across 120 generated tests.)
No sub-models (for neither hardware nor software) produces 17 hardware configurations (across 17 generated tests.)
So what's happening here? Am I looking at the PICT results wrong, or at the documentation wrong? Or could the documentation be improved by changing this:
{ PLATFORM, CPUS, RAM, HDD } @ 3
... to this:
{ PLATFORM, CPUS, RAM, HDD } @ 2
—
Reply to this email directly, view it on GitHub<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2Fpict%2Fissues%2F95&data=05%7C01%7Cjacekcz%40microsoft.com%7C0c93a70f7b894417cb5308dad4822483%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638055952673283035%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=lD5n1R1edh2wrzY17%2B%2F5lgQT6NkfVLbGbAuQPCK85xQ%3D&reserved=0>, or unsubscribe<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FACFRMV7VCBDJXZVOZ6NRG2LWLIPP5ANCNFSM6AAAAAASSCVR5M&data=05%7C01%7Cjacekcz%40microsoft.com%7C0c93a70f7b894417cb5308dad4822483%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638055952673283035%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=GPUYiEDOUcBpP93MOa3BIudtLRsnpdfywThNO0anIBM%3D&reserved=0>.
You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>
|
Thank you for your response. Regarding the part "The trend is going to be more visible with larger models. E.g. add a few more parameters to it to see the real difference." - I tried that today and I'm afraid I don't follow. I've included an example model below, and some commented-out sub-models which I also tried. None of the sub-model variations produced fewer hardware configurations than my model without the sub-models. (No sub-models produced 20 hardware configurations, with var7 to var13). If, by chance, you could show me a model which shows the real difference, please do. I'm interested because as it stands, it seems that I don't understand sub-models. Maybe I'm not the only one. PLATFORM: x86, x64, arm { PLATFORM, CPUS, RAM, HDD } @ 3 #{ PLATFORM, CPUS, RAM, HDD , var7, var8, var9, var10, var11, var12, var13 } @ 3 |
Let’s simplify the model a bit to see the difference. Let’s use this:
PLATFORM: x86, x64, arm
CPUS: 1, 2, 4
RAM: 1GB, 4GB, 64GB
HDD: SCSI, IDE
OS: Win7, Win8, Win10
Browser: Edge, Opera, Chrome, Firefox
APP: Word, Excel, Powerpoint
{ PLATFORM, CPUS, RAM, HDD } @ 2
When you produce an all pairs (pict model.txt /o:2) output on that model, I see the following:
188 combinations of values to cover, 40 generated tests, 11 combinations of the first four parameters.
When I remove the sub-model, I get:
Still 188 combinations of values to cover: no surprise here.
17 generated tests each
17 unique combinations of the first four parameters.
The point here is that you have some fixed number of combinations to cover (188 in this case), but the algorithm has a choice of how to cover. Sometimes it makes sense to trade off an increase in the absolute number of tests for a decrease of uniqueness of a subset. I use the example where some parameters are more costly to combine together than others. In the example above, I might like to set up 11 unique hardware combinations to run 40 tests than 17 unique hardware configs to run one test each.
Does this help?
From: Dan Tripp ***@***.***>
Date: Monday, December 5, 2022 at 8:33 AM
To: microsoft/pict ***@***.***>
Cc: Jacek Czerwonka ***@***.***>, Comment ***@***.***>
Subject: Re: [microsoft/pict] Documentation on sub-models (Issue #95)
Thank you for your response. Regarding the part "The trend is going to be more visible with larger models. E.g. add a few more parameters to it to see the real difference." - I tried that today and I'm afraid I don't follow. I've included an example model below, and some commented-out sub-models which I also tried. None of the sub-model variations produced fewer hardware configurations than my model without the sub-models. (No sub-models produced 20 hardware configurations, with var7 to var13). If, by chance, you could show me a model which shows the real difference, please do. I'm interested because as it stands, it seems that I don't understand sub-models. Maybe I'm not the only one.
PLATFORM: x86, x64, arm
CPUS: 1, 2, 4
RAM: 1GB, 4GB, 64GB
HDD: SCSI, IDE
separator: zzz
OS: Win7, Win8, Win10
Browser: Edge, Opera, Chrome, Firefox
APP: Word, Excel, Powerpoint
var7: x, y, z
var8: x, y, z
var9: x, y, z
var10: x, y, z
var11: x, y, z
var12: x, y, z
var13: x, y, z
{ PLATFORM, CPUS, RAM, HDD } @ 3
{ OS, Browser } @ 2
#{ PLATFORM, CPUS, RAM, HDD , var7, var8, var9, var10, var11, var12, var13 } @ 3
#{ OS, Browser , var7, var8, var9, var10, var11, var12, var13 } @ 2
—
Reply to this email directly, view it on GitHub<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2Fpict%2Fissues%2F95%23issuecomment-1337676053&data=05%7C01%7Cjacekcz%40microsoft.com%7Caa79165923294e608a0908dad6de657a%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638058547926983851%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=feo4XghOyNzN9xvzgC46LDMApyzJq3EM%2BxWcMzigVsQ%3D&reserved=0>, or unsubscribe<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FACFRMVY4M56RJ7EXYLUKSQ3WLYKMHANCNFSM6AAAAAASSCVR5M&data=05%7C01%7Cjacekcz%40microsoft.com%7Caa79165923294e608a0908dad6de657a%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638058547926983851%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=1q8pfDtcttajp4hyXWYWrItenLaA5n%2BXDwf%2FdVkBCVI%3D&reserved=0>.
You are receiving this because you commented.Message ID: ***@***.***>
|
That cleared it up. Thank you. If I made a pull request that adds some of this discussion to pict.md, would you consider it? |
Right on. Now there's something else I'd like to ask, so that I can get the details of this PR correct: These are the 10 that I see: To my surprise, if I use this model below (with no sub-model and no non-hardware parameters) then I see 11: What do you think? |
That is by-design. Essentially, packing n-way combinations with as small a number of tests is a “hard problem” for which PICT and other tools use heuristics. These are deterministic but they are dependent on initial conditions. Sometimes the algorithm is a bit more lucky and packs all desired combinations in 10 rows, sometimes 11 (or more).
To see this in practice, you can run PICT with /r which will seed the “initial conditions” with a random value for each run. See how many tests you get in each (/s is very good to use in those cases):
./pict model.txt /r /s
…--Jacek
From: Dan Tripp ***@***.***>
Date: Monday, December 5, 2022 at 3:55 PM
To: microsoft/pict ***@***.***>
Cc: Jacek Czerwonka ***@***.***>, Comment ***@***.***>
Subject: Re: [microsoft/pict] Documentation on sub-models (Issue #95)
Right on. Now there's something else I'd like to ask, so that I can get the details of this PR correct:
You wrote "188 combinations of values to cover, 40 generated tests, 11 combinations of the first four parameters." But I see 10, not 11.
These are the 10 that I see:
arm 1 64GB IDE
arm 2 1GB SCSI
arm 2 4GB IDE
arm 4 64GB IDE
x64 1 4GB SCSI
x64 2 64GB IDE
x64 4 1GB SCSI
x86 1 1GB IDE
x86 2 64GB SCSI
x86 4 4GB IDE
To my surprise, if I use this model below (with no sub-model and no non-hardware parameters) then I see 11:
PLATFORM: x86, x64, arm
CPUS: 1, 2, 4
RAM: 1GB, 4GB, 64GB
HDD: SCSI, IDE
What do you think?
—
Reply to this email directly, view it on GitHub<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2Fpict%2Fissues%2F95%23issuecomment-1338380823&data=05%7C01%7Cjacekcz%40microsoft.com%7C7349f24c451441e4d9bd08dad71c3e69%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638058813553916121%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2BX%2FBHct0QULzrNhw32jLq0dessdXODBEP2XbTsZIUqw%3D&reserved=0>, or unsubscribe<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FACFRMV4VFZ6LEJIZEMFYFZ3WLZ6IPANCNFSM6AAAAAASSCVR5M&data=05%7C01%7Cjacekcz%40microsoft.com%7C7349f24c451441e4d9bd08dad71c3e69%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638058813553916121%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=4RmggVQ8ZI%2FfEhJgIrJ1eCTIcOcwWFUH9UM6Fzqo39Q%3D&reserved=0>.
You are receiving this because you commented.Message ID: ***@***.***>
|
Okay, I made a PR. I hope it helps. I should get permission from my employer before I click "agree" on the Contributor License Agreement. I'll try to do that later this week. |
Closed by #112, I think. |
There's something that I don't understand in the documentation in pict.md, where it says:
"Placing all hardware parameters into one sub-model produces fewer distinct hardware configurations and potentially lowers the cost of testing."
But the example shown (with { PLATFORM, CPUS, RAM, HDD } @ 3) seems to produce more distinct hardware configurations. Not fewer.
By my count:
{ PLATFORM, CPUS, RAM, HDD } @ 3 produces 28 hardware configurations (across 336 generated tests.)
{ PLATFORM, CPUS, RAM, HDD } @ 2 produces 10 hardware configurations (across 120 generated tests.)
No sub-models (for neither hardware nor software) produces 17 hardware configurations (across 17 generated tests.)
So what's happening here? Am I looking at the PICT results wrong, or at the documentation wrong? Or could the documentation be improved by changing this:
{ PLATFORM, CPUS, RAM, HDD } @ 3
... to this:
{ PLATFORM, CPUS, RAM, HDD } @ 2
The text was updated successfully, but these errors were encountered: