newsletter "source" is sometimes populated, sometimes not

[stlhood]

1. When unsubscribing-all (via the Preference Center), the user is presented with a form from which they can indicate the reason they unsubscribed (including a free text option). CTMS is not populating this data in the corresponding field ("unsubscribe_reason") on the "email" record.
2. CTMS is not populating the "source" field on the "newsletter" field. Basket should be providing this.

[jwhitlock]

When unsubscribing-all (via the Preference Center), the user is presented with a form from which they can indicate the reason they unsubscribed (including a free text option). CTMS is not populating this data in the corresponding field ("unsubscribe_reason") on the "email" record.

I tried selecting "Remove me from all the subscriptions on this page" from the Preference center. This marked all the newsletters as unsubscribed almost immediately. The next page asked for a reason. It seemed to take more than a minute, but the "unsubscribe_reason" field was eventually set.

I suspect this is because get_lock is called, which adds a delay (default 60 seconds) to prevent rapid writes to the same record in Salesforce. The lock prevents the second write from taking place immediately. In CTMS, the update timestamps showed the unsubscribe at 20:19:33, and the reason populated at 20:23:02, 3 1/2 minutes later.

There's a per-subscription unsubscribe in CTMS, but because these are done as two requests, we can't actually associate the unsubscribe reason with the newsletters that were just unsubscribed. Maybe after the transition.

CTMS is not populating the "source" field on the "newsletter" field. Basket should be providing this.

I was able to reproduce this, using the subscription form for the mozilla-foundation newsletter on the front page of www-dev. The source URL was set on one code path, and basket PR 705 adds it in the other code path. @pmac reviewed and merged, and now the source URL is set.

[stlhood]

Further testing confirms that unsubscribe_reason is in fact being populated in both CTMS and Acoustic, just sometimes with a delay.

Regarding the newsletter source filed, note that testing this evening shows that it is being populated for the firefox newsletter ("mozilla-and-you") but not for any others that were tested. Is this expected until the latest commit is deployed to Basket dev?

[jwhitlock]

The basket code has been deployed to basket-dev, so you are seeing the latest.

Maybe the signup URL is not sent for some code paths? Can you see a pattern in how you signed up for a newsletter and if the source is set in CTMS?

[pmac]

depends if the form sends source_url as well. are you using the same signup form every time?

[jwhitlock]

There's one test (2.6) that uses curl to subscribe to a MoFo-relevant newsletter, and doesn't provide a source. This is an instance where the source URL will not be sent, since it is not in the POST body and curl doesn't set a referrer header like a browser.

[stlhood]

Ah ha, then that explains it. Thank you, and sorry for the wild goose chase on this one.

[stlhood]

FYI from my perspective we can close this.

[jwhitlock]

I don't think there is additional work in CTMS to set the source URL.

I think we could populate the unsubscribe reason on the basket side, as well as update the format and language of existing newsletters. I've added these ideas in mozmeao/basket#698.

[jwhitlock]

There was also an earlier issue for the source URL not matching between SFDC and CTMS, that is tracked in mozmeao/basket#699