Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CPM filterlist experiment #3086

Merged
merged 17 commits into from
Aug 19, 2024
Merged

CPM filterlist experiment #3086

merged 17 commits into from
Aug 19, 2024

Conversation

muodov
Copy link
Member

@muodov muodov commented Aug 8, 2024

Task/Issue URL: https://app.asana.com/0/1201621853593513/1207940672701584/f
Tech Design URL:
CC:

Description:
This adds a CPM filterlist experiment as described in Asana

Steps to test this PR:

  1. Check out this branch and the corresponding BSK branch with the same name
  2. Build
  3. Enable Autoconsent logs in Debug menu, make sure that CPM feature is enabled in user settings
  4. Point to a custom config with the new CPM subfeature:
        "autoconsent": {
            "state": "enabled",
            "features": {
                "filterlistExperiment": {
                    "state": "enabled",
                    "rollout": {
                        "steps": [
                            {
                                "percent": 100
                            }
                        ]
                    }
                }
            }
        },
  1. Search for "buy shoes" on DDG. Verify in logs that the experiment cohort is picked ("fc" for control, or "ft" for test)
  2. Click on some ads, verify that m_mac_ad_click_detected pixel has the corresponding parameter value ("1" for test, "0" for control)
  3. In Debug menu, select Reset Data -> Reset CPM Experiment Cohort to clear the cohort. Restart the app.
  4. Repeat resetting until you get a different cohort. Verify that pixel parameters change accordingly.
  5. Disable the subfeature in config, restart the app. Verify that the pixel parameter is NOT set in this case.

Definition of Done:


Internal references:

Pull Request Review Checklist
Software Engineering Expectations
Technical Design Template
Pull Request Documentation

@muodov muodov changed the title Max/cpm-ad-experiment CPM filterlist experiment Aug 8, 2024
@muodov muodov requested a review from federicocappelli August 8, 2024 14:14
@federicocappelli
Copy link
Member

@muodov could you please remove the local BSK and point the package to the specific branch and commit it?
Screenshot 2024-08-09 at 09 44 18

Copy link
Contributor

github-actions bot commented Aug 9, 2024

Warnings
⚠️ PR has more than 500 lines of code changing. Consider splitting into smaller PRs if possible.

Generated by 🚫 dangerJS against 808f6c3

@muodov muodov force-pushed the max/cpm-ad-experiment branch from 52b7718 to 4c7e831 Compare August 9, 2024 12:51
@muodov muodov force-pushed the max/cpm-ad-experiment branch from 4c7e831 to 0c77968 Compare August 9, 2024 12:52
@@ -360,19 +360,9 @@ jobs:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v4
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you explain these changes? Seems strange to modify the CI in main

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This CI check is verifying that the autoconsent JS bundle matches the version pinned in the dependencies. This is mainly to catch when we update package.json, but do not actually commit the recompiled JS bundle.
In this case, we're actually doing exactly that: shipping a custom version that is not released as a proper library version. I removed this check so that it doesn't fail every build while the experiment is active

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Understood, do we have a task scheduled for re-activating this check?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fantastic thanks, then the PR LGTM if we agree that the monitoring part is enough.
You could merge BSK, release a new version (patch version bump I think) and update the branch here in the meantime.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @federicocappelli! It's great to have this change approved. Once we the JS review and the monitoring discussion are resolved, I'll do a final merge.

@muodov
Copy link
Member Author

muodov commented Aug 15, 2024

@federicocappelli I've added the parameter to the breakage pixel too. Could you check if it looks ok?

muodov added a commit to duckduckgo/BrowserServicesKit that referenced this pull request Aug 19, 2024
<!--
Note: This checklist is a reminder of our shared engineering
expectations.
-->

Please review the release process for BrowserServicesKit
[here](https://app.asana.com/0/1200194497630846/1200837094583426).

**Required**:

Task/Issue URL:
https://app.asana.com/0/1201621853593513/1207940672701584/f
iOS PR: 
macOS PR: duckduckgo/macos-browser#3086
What kind of version bump will this require?: Major/Minor/Patch

**Optional**:

Tech Design URL:
CC:

**Description**:
See the macos PR for details

<!--
Tagging instructions
If this PR isn't ready to be merged for whatever reason it should be
marked with the `DO NOT MERGE` label (particularly if it's a draft)
If it's pending Product Review/PFR, please add the `Pending Product
Review` label.

If at any point it isn't actively being worked on/ready for
review/otherwise moving forward (besides the above PR/PFR exception)
strongly consider closing it (or not opening it in the first place). If
you decide not to close it, make sure it's labelled to make it clear the
PRs state and comment with more information.
-->

**Steps to test this PR**:
1.
1.

<!--
Before submitting a PR, please ensure you have tested the combinations
you expect the reviewer to test, then delete configurations you *know*
do not need explicit testing.

Using a simulator where a physical device is unavailable is acceptable.
-->

**OS Testing**:

* [ ] iOS 14
* [ ] iOS 15
* [ ] iOS 16
* [ ] macOS 10.15
* [ ] macOS 11
* [ ] macOS 12

---
###### Internal references:
[Software Engineering
Expectations](https://app.asana.com/0/59792373528535/199064865822552)
[Technical Design
Template](https://app.asana.com/0/59792373528535/184709971311943)
@muodov muodov marked this pull request as ready for review August 19, 2024 08:01
Copy link
Member

@federicocappelli federicocappelli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The use of singletons is not how we want to do this but it's ok for an experiment.
LGTM

@muodov muodov merged commit 292fa4f into main Aug 19, 2024
20 checks passed
@muodov muodov deleted the max/cpm-ad-experiment branch August 19, 2024 09:35
@muodov muodov mentioned this pull request Nov 12, 2024
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants