Create and start tracking product metrics

[JnBrymn]

Context

Inspired by a couple of YC's How to build a startup videos (I think 4 and 6) we need to start tracking metrics about usage so that we know how healthy our product and community are.

I propose we find metrics with these qualities:

• Easy to implement.
• Simple to understand.
• Persistent, meaning that we can implement them now and they will still be useful as we change product and introduce new things.
• Track engagement and retention of members

I propose these specific metrics:

Engagement:

• Number of Penny Chats created.
• Number of Followups created.
• Number of visits to Chats page.
• Number of visits to Profile page.
• Across all above metrics, count of unique users.

†These metrics should exclude the "expected people" like Nick and John

Retention:

This needs to be something like "for all users that did anything of the above actions in week 1, how many of them did something in week 2, week 3, ... ". This is important, but harder to measure because the data is so sparse. The richest stream of data is Slack, but I don't want to test their platform or our community, I want to test our platform. The engagements above are too sparse to matter much. Probably the best metrics to track are the profile and chat page views. We need to make sure that Google Analytics is set up to record this correctly and to present it as a retention graph. (I'm pretty sure they do this.)

TODO

• Settle on a handful of metrics that are meaningful and easy to track.
• Write code to start tracking the metrics. (Might not be necessary.)
• Create a report generator that summarized important metrics into #penny-labs and also links out to other metrics like our Google Analytics.
  • Make sure to look into how to exclude internal traffic https://support.google.com/analytics/answer/1034840?hl=en

[mrshwah]

I'm totally in support of this, but some of these metrics might be better suited for Google Analytics, for example the visits to the profile and chats page. So we should figure out which ones we need to write ourselves and which ones we should track in GA.

[JnBrymn]

Yeah, I agree. Updated ticket accordingly.

[ghost]

RE: Google Analytics

For my websites I've found a simple way to filter out any data coming from a set of IP addresses, designed to make it so that for example your company's internal access of a site doesn't drastically change the data. This guide may be good to note for excluding certain users.

Not necessarily attached to the product 100%, but some other good things to maybe keep track of for doing consumer research inside the Penny U Slack itself:

• What channels are Penny Chats sent to most often?
• What keywords are being used the most in active channels?
• What percentage of users make up active (in a certain timeframe) Penny Chat participants?

These would most likely need to be tracked in the bot itself which could make for more work, but I think the above 3 could strengthen the knowledge of the current demographic, where you could strengthen it, and where it can be expanded.

[JnBrymn]

Nice @deadbender! I've added the link to the description.