Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add diffing for ABT Experiments in real-time RC #5623

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Conversation

qdpham13
Copy link
Contributor

@qdpham13 qdpham13 commented Jan 3, 2024

Adds diffing logic between experiments fetched via real-time and experiments active on the device. The config keys that have been affected by the differences will be added to ConfigUpdate. Testing below:

Created experiments with https://docs.google.com/document/d/1MXxUg1JSBbh3m_Qy4tGPrGs3-Nh8FAwKmg5ZWOKQ3J8/edit

Add experiments to configs using Management API, example: https://rpc.corp.google.com/rpc/builder?rpcId=15352346049097272373

2.1) Add/Remove Experiment to Config Key.
2.2) Replace experiment in Config Key
2.3) Create new param with experiment attached

See changes in SDK

Copy link
Contributor

github-actions bot commented Jan 3, 2024

Release note changes

No release note changes were detected. If you made changes that should be
present in the next release, ensure you've added an entry in the appropriate
CHANGELOG.md file(s).

@google-oss-bot
Copy link
Contributor

1 Warning
⚠️ Did you forget to add a changelog entry? (Add the 'no-changelog' label to the PR to silence this warning.)

Generated by 🚫 Danger

@google-oss-bot
Copy link
Contributor

google-oss-bot commented Jan 3, 2024

Coverage Report 1

Affected Products

  • firebase-config

    Overall coverage changed from 84.10% (08eeaf6) to 84.25% (8077b60) by +0.15%.

    FilenameBase (08eeaf6)Merge (8077b60)Diff
    ConfigContainer.java94.29%94.48%+0.19%

Test Logs

  1. https://storage.googleapis.com/firebase-sdk-metric-reports/0Z41pvizMo.html

Copy link
Contributor

github-actions bot commented Jan 3, 2024

Unit Test Results

  40 files   -    908    40 suites   - 908   1m 0s ⏱️ - 30m 36s
311 tests  - 4 867  311 ✔️  - 4 846  0 💤  - 21  0 ±0 
634 runs   - 9 807  634 ✔️  - 9 765  0 💤  - 42  0 ±0 

Results for commit 8e65687. ± Comparison against base commit a5d0f0d.

♻️ This comment has been updated with latest results.

@google-oss-bot
Copy link
Contributor

google-oss-bot commented Jan 3, 2024

Size Report 1

Affected Products

  • firebase-config

    TypeBase (08eeaf6)Merge (8077b60)Diff
    aar108 kB109 kB+603 B (+0.6%)
    apk (aggressive)265 kB266 kB+396 B (+0.1%)
    apk (release)4.81 MB4.81 MB+504 B (+0.0%)

Test Logs

  1. https://storage.googleapis.com/firebase-sdk-metric-reports/enO23zUoWw.html

@google-oss-bot
Copy link
Contributor

google-oss-bot commented Jan 3, 2024

Startup Time Report 1

Note: Layout is sometimes suboptimal due to limited formatting support on GitHub. Please check this report on GCS.

Notes

Startup Times

  • fire-rc

    DeviceStatisticsDistributions
    oriole-32
    Percentile08eeaf68077b60DiffSignificant (?)
    p10528 ±866 μs448 ±690 μs-79.3 μs (-15.0%)NO
    p25550 ±903 μs480 ±734 μs-70.7 μs (-12.9%)NO
    p50607 ±995 μs532 ±788 μs-74.7 μs (-12.3%)NO
    p75700 ±1140 μs624 ±886 μs-76.3 μs (-10.9%)NO
    p90820 ±1325 μs782 ±1109 μs-38.5 μs (-4.7%)NO

    19 test runs in comparison
    CommitTest Runs
    08eeaf6
    • 2024-01-22_20:42:49.349637_yVAq
    • 2024-01-22_20:42:49.349678_BYEY
    • 2024-01-22_20:42:49.349686_pPFT
    • 2024-01-22_20:42:49.349692_eZyB
    • 2024-01-22_20:42:49.349699_Etkn
    • 2024-01-22_20:42:49.349704_ZPSC
    • 2024-01-22_20:42:49.349708_UcYt
    • 2024-01-22_20:42:49.349715_AKSW
    • 2024-01-22_20:42:49.349719_QgUu
    8077b60
    • 2024-01-22_21:50:27.122037_LbyW
    • 2024-01-22_21:50:27.122075_lccw
    • 2024-01-22_21:50:27.122087_vaLw
    • 2024-01-22_21:50:27.122096_GsuN
    • 2024-01-22_21:50:27.122104_cyCV
    • 2024-01-22_21:50:27.122112_xEsa
    • 2024-01-22_21:50:27.122120_CfEb
    • 2024-01-22_21:50:27.122137_zgwX
    • 2024-01-22_21:50:27.122144_JQyv
    • 2024-01-22_21:50:27.122151_IUNe
    redfin-30
    Percentile08eeaf68077b60DiffSignificant (?)
    p10551 ±606 μs417 ±500 μs-133 μs (-24.2%)NO
    p25626 ±667 μs474 ±592 μs-152 μs (-24.3%)NO
    p50785 ±786 μs567 ±732 μs-218 μs (-27.8%)NO
    p751.06 ±0.9 ms720 ±937 μs-338 μs (-31.9%)NO
    p901.57 ±1 ms902 ±1183 μs-663 μs (-42.4%)NO

    19 test runs in comparison
    CommitTest Runs
    08eeaf6
    • 2024-01-22_20:42:49.349637_yVAq
    • 2024-01-22_20:42:49.349678_BYEY
    • 2024-01-22_20:42:49.349686_pPFT
    • 2024-01-22_20:42:49.349692_eZyB
    • 2024-01-22_20:42:49.349699_Etkn
    • 2024-01-22_20:42:49.349704_ZPSC
    • 2024-01-22_20:42:49.349708_UcYt
    • 2024-01-22_20:42:49.349715_AKSW
    • 2024-01-22_20:42:49.349719_QgUu
    8077b60
    • 2024-01-22_21:50:27.122037_LbyW
    • 2024-01-22_21:50:27.122075_lccw
    • 2024-01-22_21:50:27.122087_vaLw
    • 2024-01-22_21:50:27.122096_GsuN
    • 2024-01-22_21:50:27.122104_cyCV
    • 2024-01-22_21:50:27.122112_xEsa
    • 2024-01-22_21:50:27.122120_CfEb
    • 2024-01-22_21:50:27.122137_zgwX
    • 2024-01-22_21:50:27.122144_JQyv
    • 2024-01-22_21:50:27.122151_IUNe
  • timeToInitialDisplay

    DeviceStatisticsDistributions
    oriole-32
    Percentile08eeaf68077b60DiffSignificant (?)
    p10203 ±7 ms206 ±2 ms+2.95 ms (+1.4%)NO
    p25210 ±7 ms213 ±2 ms+2.57 ms (+1.2%)NO
    p50218 ±7 ms221 ±2 ms+2.87 ms (+1.3%)NO
    p75226 ±8 ms229 ±2 ms+3.16 ms (+1.4%)NO
    p90234 ±9 ms242 ±5 ms+7.54 ms (+3.2%)NO

    19 test runs in comparison
    CommitTest Runs
    08eeaf6
    • 2024-01-22_20:42:49.349637_yVAq
    • 2024-01-22_20:42:49.349678_BYEY
    • 2024-01-22_20:42:49.349686_pPFT
    • 2024-01-22_20:42:49.349692_eZyB
    • 2024-01-22_20:42:49.349699_Etkn
    • 2024-01-22_20:42:49.349704_ZPSC
    • 2024-01-22_20:42:49.349708_UcYt
    • 2024-01-22_20:42:49.349715_AKSW
    • 2024-01-22_20:42:49.349719_QgUu
    8077b60
    • 2024-01-22_21:50:27.122037_LbyW
    • 2024-01-22_21:50:27.122075_lccw
    • 2024-01-22_21:50:27.122087_vaLw
    • 2024-01-22_21:50:27.122096_GsuN
    • 2024-01-22_21:50:27.122104_cyCV
    • 2024-01-22_21:50:27.122112_xEsa
    • 2024-01-22_21:50:27.122120_CfEb
    • 2024-01-22_21:50:27.122137_zgwX
    • 2024-01-22_21:50:27.122144_JQyv
    • 2024-01-22_21:50:27.122151_IUNe
    redfin-30
    Percentile08eeaf68077b60DiffSignificant (?)
    p10247 ±4 ms271 ±3 ms+24.0 ms (+9.7%)YES
    p25252 ±4 ms278 ±4 ms+25.6 ms (+10.2%)YES
    p50260 ±4 ms285 ±5 ms+25.0 ms (+9.6%)YES
    p75269 ±5 ms295 ±6 ms+25.8 ms (+9.6%)MAYBE
    p90278 ±6 ms311 ±11 ms+33.2 ms (+12.0%)MAYBE

    19 test runs in comparison
    CommitTest Runs
    08eeaf6
    • 2024-01-22_20:42:49.349637_yVAq
    • 2024-01-22_20:42:49.349678_BYEY
    • 2024-01-22_20:42:49.349686_pPFT
    • 2024-01-22_20:42:49.349692_eZyB
    • 2024-01-22_20:42:49.349699_Etkn
    • 2024-01-22_20:42:49.349704_ZPSC
    • 2024-01-22_20:42:49.349708_UcYt
    • 2024-01-22_20:42:49.349715_AKSW
    • 2024-01-22_20:42:49.349719_QgUu
    8077b60
    • 2024-01-22_21:50:27.122037_LbyW
    • 2024-01-22_21:50:27.122075_lccw
    • 2024-01-22_21:50:27.122087_vaLw
    • 2024-01-22_21:50:27.122096_GsuN
    • 2024-01-22_21:50:27.122104_cyCV
    • 2024-01-22_21:50:27.122112_xEsa
    • 2024-01-22_21:50:27.122120_CfEb
    • 2024-01-22_21:50:27.122137_zgwX
    • 2024-01-22_21:50:27.122144_JQyv
    • 2024-01-22_21:50:27.122151_IUNe

  1. https://storage.googleapis.com/firebase-sdk-metric-reports/7kPrPHc78L/index.html

@qdpham13 qdpham13 requested a review from danasilver January 10, 2024 23:26
for (int j = 0; j < affectedKeys.length(); j++) {
String key = affectedKeys.getString(j);
JSONObject experimentsCopy = new JSONObject(experiment.toString());
// Removing `affectedParameterKeys` because its values never come in the same order which
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thinking: and because we're diffing the metadata w.r.t. the key(s) itself so we don't need to diff the key

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah that makes sense, removed this

Copy link
Contributor

@danasilver danasilver left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will want to add a changelog entry too, but can happen in a followup PR

@@ -189,6 +193,38 @@ public boolean equals(Object o) {
return containerJson.toString().equals(that.toString());
}

/** Creates a map where the key is the config key and the value if the experiment description. */
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/** Creates a map where the key is the config key and the value if the experiment description. */
/** Creates a map where the key is the config key and the value is the experiment description. */

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated

@@ -189,6 +193,38 @@ public boolean equals(Object o) {
return containerJson.toString().equals(that.toString());
}

/** Creates a map where the key is the config key and the value if the experiment description. */
private Map<String, JSONObject> createExperimentsMap(JSONArray allExperiments)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this could be static? Though alternatively it could be nice to make it have a similar signature to createRolloutParameterKeyMap work as an instance method so we call this.createExperimentsMap() and other.createExperimentsMap()

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated to just use instance vars

@@ -138,23 +139,99 @@ public void getChangedParams_sameP13nMetadata_returnsEmptySet() throws Exception
}

@Test
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice tests for this

@qdpham13 qdpham13 requested a review from danasilver January 26, 2024 22:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants