Session User Engagement Report #60
Comments
|
the comrade writes very right things ! |
|
Quote from Google Play Console's definition of user metrics:
It's important to note that Google's official definition includes some wording that could be confusing. Active users refers to users who 'use the device', whereas Monthly Active Users are described as users who 'open the app'. |
|
Update 2023-12-21
|
|
Update:
Notes:
|
|
Great |
|
Last month (Jul 2024) was a difficult one, with recent bugs causing a 15% drop in MAU and a 25% drop in DAU. The good news is that those bugs appear to have been fixed and new releases have been rolled out. Let's see how much we can recover next month. Edit: According to service node data, MAU and DAU for August 2024 remain almost the same as July 2024. In other words, there hasn't been much recovery since the recent service node bug was fixed. Apparently, it is much easier to lose users than to gain them. There are some discrepancies in user numbers between the August 18th Session official blog announcement and the metrics derived by service nodes. Here is a clarify for the discrepancy from Kee. |
AppBrain data also shows a significant rating decrease in the last few months (Mar 2024 to Jul 2024) for Session Android, indicating that more work needs to be done. Let's see if the upcoming new release with the user onboarding overhaul helps, otherwise we might need some more bug fix releases. |
|
Hey @venezuela01, we recently encountered several issues with the Service Node "oxen storage server" package, which affected user interactions with communities, the Session file server, and even the ability to send 1-1 messages. This period of instability began in early July and was resolved last week. Understandably, many users were frustrated by these disruptions and left 1-star reviews, which we believe is the primary reason for our app's lower rating recently. We are now focused on improving and stabilizing the core experience of the app to restore and surpass our previous rating. This message describes the state of the network shortly after the Oxen storage server 2.7.0 mandatory Service Node upgrade (MSNU), we have now deployed client and server side fixes in storage server 2.8.0 and initiated a new MSNU.
|
@KeeJef Thank you for the detailed information and for your diligent work in delivering the fix. Looking ahead, I believe there are two key areas that could enhance the long-term stability of the project:
I hope we can add these to our wish list for the next 6 to 18 months. |
|
On August 18, 2024, the official Session blog claimed Over 1 million people are using Session. I apologize for saying this, but I'm very confused and not very convinced that this number reflects reality. In my opinion, the closest we've been to 1 million users was in April 2024, where the 90% confidence interval of monthly active users was between 923k to 1 million. However, due to recent network instability causing user churn, the 90% confidence interval of Session MAU has dropped to 776k to 815k. I'm very disappointed that the Session official blog decided to publish an inflated number at this specific time. It's not a very wise strategy. I wish they took their methodology and metrics more seriously. |
|
Hey @venezuela01, there are a few things to discuss here that might help clarify the announcement. Firstly, our measurement of MAUs does not rely solely on the count of configuration/regular messages containing unique Session IDs from a selection of Service Nodes over a period of time. Instead, we synthesize data from various sources, including Service Nodes, the Google Play Store, the Apple Store, GitHub, F-Droid, and others. Why do we do this? Because there are several issues with how different clients post configuration messages to Service Nodes. For example, last week we discovered that Android clients failed to post configuration messages to their swarm when creating new accounts in some cases. This issue was present in all pre-onboarding releases. Other similar cases have been found on iOS and Android linked devices recently, which caused clients not to update their configuration messages at all when multiple devices were linked. While we are actively resolving these issues as they arise, we have not yet confirmed the absolute reliability of configuration messages. Without a high level of trust in configuration messages, it's possible to miss a significant number of active users if we rely solely on Service Node measurements. For instance, users who are active only in communities or closed groups/groups might be undercounted, as closed groups can share a single "Session ID" from the perspective of Service Nodes. Additionally, users who primarily send messages to others without receiving timely responses might also be underrepresented. Given this, our primary source for MAU data right now, supplemented by other measures, is the app stores. Google Play provides a statistic called "Monthly Active Users," defined as "the number of users who opened your app in a 28-day period." We use this measure for Google Play, while the Apple Store provides a similar statistic called "Active Last 30 Days," defined as "the total number of devices with at least one session within 30 days of the selected day." The combination of these sources, along with additional measures, produces an aggregate MAU count, which currently exceeds 1 million. |
@KeeJef Thanks for your detailed response. Firstly, I apologize for my misaccusation regarding the recent official announcement. Apparently, there are more subtle details than I knew, and I had blamed the announcement based on incomplete information. My fault. I'm very sorry. I apologize. Your explanation is helpful. While it's not completely clear if the gap can be fully attributed to the factors you pointed out, at least it's clear enough to show that my previous accusation was unfair. To eliminate similar confusion and misunderstandings in the future, may I request a few improvements to the Session user metrics announcement scheme?
On the other hand, I am inspired by your discovery and found some room for improvement in service node based measurements as well. As you pointed out, configuration message counts might not be completely reliable, which is crucial for tracking MAU. However, regular message counts and message storage size are supposed to be more reliable because if their numbers decrease, it's clear evidence of lower user activity. By tracking regular message counts, we can draw the trend line of bi-weekly active users. Similarly, we can draw the trend line of message storage size on the server. When we compare these two trend lines with other metrics, like MAU, we can see if they are correlated. If one of them is going up and the other is going down, then it's an indication that something might be wrong and needs investigation. Update: I looked back at my cached daily reports and found that the DAU metrics showed an obvious drop from July 15th to July 31st, 2024, which has not yet recovered as of today (Aug 20th). My DAU was calculated using regular messages rather than configuration messages, so it shouldn't be affected by configuration message bugs. My observation is that DAU based on service node regular messages is a very helpful indicator of user churn. I think it would be a waste of knowlege if we interpret these metrics as solely marketing tools. They are more than that—they are diagnostic tools and growth tools. Thank you very much. |
|
Congratulations on another incredible milestone of Session downloads! The recent tweet from the official Session Token Twitter account states:
That's a fantastic achievement, and it speaks volumes about the amazing work the Session team is doing. To ensure this success continues, I was curious about the phrasing used in the post. Could it be even more accurate to say 'Session has been downloaded over 13 million times'? This phrasing accounts for the fact that some users may have downloaded Session multiple times across different versions. While celebrating this milestone, it's also important to consider how these download numbers relate to user retention. For example, let's imagine two interpretations:
Understanding which interpretations is closer to reality is crucial for assessing Session's growth and making informed decisions. It's also worth noting that investors often prioritize retention as a key metric. Ultimately, being precise and transparent about our metrics is a reflection of intellectual honesty. This commitment to accuracy helps us build trust with our community and stakeholders, fosters a culture of clear communication, and allows us to make better decisions. I'm always happy to engage in user activity analysis. If there's anything I can volunteer for in the future to make clearer data reports, please let me know. Thanks again for the hard work! |
|
Update: User reviews on the Google Play Store have increased significantly over the past 5 months. Kudos to the team for their hard work and dedication! 
|
|
Thanks @venezuela01 still working on bringing things up even more, but the teams working on Session have definitely made good progress |
Session User Engagement Report
Edit: read #60 (comment) for latest update.
Introduction
This report provides a detailed analysis of user activity statistics sourced from the
storage.dbof the Oxen Storage Server, with data fetched in the end of October 2023. The report delivers an exhaustive review of user engagement, showcasing metrics for Monthly and Biweekly Active Users at both the individual server and the network-wide scale. It also emphasizes the distribution of user activity, indicating that the top 15% of the most active users account for more than 80% of the message traffic, thus demonstrating a pronounced manifestation of the Pareto distribution.Data Preparation
storage.dbwas procured.pubkey_to_swarm_space_position()function, in conjunction with the methodology outlined in Oxen Storage Server Issue oxen-io/session-android#470, was employed to filter out outdated messages. This step is essential to bypass a known bug and to avoid overestimating the number of active users on a single storage server.Monthly Active User (MAU)
On a Single Storage Server
By ensuring that no obsolete messages were included and by conducting a thorough data cleanup, we were able to tally distinct owner IDs in the message table. Our server interacted with 2,961 active users for the month.
Network-Wide Estimation
To approximate the Monthly Active Users across the entire network, we calculated the swarm space coverage ratio of our storage server.
The ratio was determined by:
swarm_space_positionusingpubkey_to_swarm_space_position()swarm_space_positionand identifying the minimum and maximum valuesswarm_space_positionby2^64, where the denominator denotes the total swarm space.Consequently, we deduced that our server covers 1/256 of the network space.
By extrapolation, we estimate the network comprises 758,016 (758k) Monthly Active Users (2,961 multiplied by 256).
Edit: Starting from 2024-02-01, we no longer use the min-max estimation for swarm width. Instead, we use the
get_service_nodesJSON-RPC call to obtain all swarm IDs of all nodes and calculate the precise swarm boundaries. The results are almost the same as those obtained through the min-max estimation approach, but the precise approach via JSON-RPC is more sensitive in detecting swarm bugs, allowing us to intervene manually when necessary.Biweekly Active User Analysis
We dive deep into biweekly active users because regular user messages have a 14-days Time-To-Live (TTL). Regular user messages were extracted by filtering messages according to their TTL value. This procedure enabled us to focus on regular user messages and to exclude configuration messages.
The refined dataset contained 283,365 messages from 1,917 users over a biweekly span.
Network-Wide Projection
Applying the same coverage ratio as before (1/256), we project that there are approximately 490,752 Biweekly Active Users network-wide. The same scaling factor is applied in the following analysis.
Pareto Distribution of User Activity
Note on Message Ownership: In the current design of Session, when a user sends a message, a copy is also sent to their own swarm; when a user receives a message, their swarm receives the message. The term 'owning' a message encompasses both sending and receiving; this adds a layer of complexity to the analysis, which we will simplify for the moment by aggregating all such activities under general user engagement.
Distribution of Message Ownership Among Users
Our analysis of user behavior within the network has revealed a pronounced imbalance: a relatively small fraction of users are engaged with a disproportionately large share of messages. This aligns with the Pareto Distribution, which suggests that a small number of individuals often account for a large portion of the effects.
The following table illustrates the cumulative percentage of active users in comparison to the cumulative percentage of messages they are engaged with:
This data indicates that in a two weeks span, about 3.5% of the most active users engage with 50% of the messages, and around 15.2% of users account for about 80% of the messaging activity. A similar distribution was noted when analyzing the data based on message storage size instead of message count.
Distribution of User Activity Across Time Buckets
To better understand how often users are active, we split two weeks' worth of messages into 14 daily groups. We tracked how many days each user engages. Then, we group users with the same number of active days together and counted number of users in each group.
The results are displayed in a Pareto chart reflecting user activity frequency.
Our findings show that, out of 491k biweekly active users, 200k (approximately 40%) were active on only one of the 14 days, 83k were active on two days, and so on. Only 15k users consistently engage with Session everyday. It's crucial to distinguish this 15k figure from the Daily Active Users (around 126k DAU) metric, which measures the number of unique users who interact with the app within a 24-hour interval, without any guarantee of their return the following day.
It is also noteworthy that with 758k monthly active users and 491k biweekly active users, there are approximately 267k users who have periods of inactivity exceeding 14 days. This raises a concern that they may miss messages due to the 14-day Time-To-Live (TTL) policy.
Daily Active Users (DAU)
We splitted messages into 14 time-based buckets and calculated the number of daily active users. After scaling up with a factor of 256, we extrapolate a DAU between 111,616 and 137,472 on a network-wide basis. On average, we estimate there are approximately 126,244 daily active users across the network.
DAU / MAU Ratio
The DAU/MAU ratio is a key performance indicator that measures user retention and engagement. For our network, this ratio is calculated as follows: 126,244 / 758,016 = 16.7%.
Comparative Analysis with Other Products
We compare the DAU/MAU ratio of Session with other products:
Session's DAU/MAU ratio is considerably lower than the industry average, potentially signaling its immaturity. This is further supported by the current average rating of the Session Android app, which stands at approximately 3.6, lagging behind the overall average app rating of 4.0 as reported by AppBrain.
Insights and Considerations
Cross-validation of User Statistics: The official number of monthly Session users is reported using a synthesis formula approach. It would be beneficial to cross-validate this method with other approaches and calibrate parameters as necessary. A discrepancy between different metrics does not necessarily mean one is incorrect. For instance, if the synthesis formula by OPTF yields a higher user count than the Oxen storage server data, it could be that many users download the Session app but struggle with account creation or finding friends to communicate with. Another possibility is that a significant percentage of users are using multiple devices, which would be counted only once when calculated using Oxen storage server data.
Community Perception vs. Official Figures: Persistent doubts among community members regarding their perception of the number of Session users compared to official figures may find some explanation in our Pareto distribution analysis. The most active 3.5% of biweekly users (approximately 17k out of 491k) account for 50% of the messages, which corresponds closely with the 15k users consistently engage everyday, yet this figure is much lower than the total MAU. Moreover, the DAU/MAU ratio for Session is notably lower than what is typical in the industry, which may contribute to the community's perception of a smaller user base.
Potential for Monetization: The concentration of activity among a small group of users suggests that we may be closer to achieving monetization than previously considered. If we target the most enthusiastic users and address their common pain points, monetization could be promissing. Assuming that the willingness to pay correlates with user engagement, the most enthusiastic 17k users, representing 3.5% of biweekly or 2.3% of monthly users, might be willing to pay $5 per month. This could potentially generate close to $1 million in annual revenue.
Openness to New Features: The same group of highly active users may also be more open to trying new features, such as making payments in Oxen. Their willingness to engage with new aspects of the platform could be crucial for the success of Oxen.
Privacy Risks with Token Changes: The significant concentration of activity among a small subset of users raises concerns about the increased risk of deanonymization if the Oxen coin is replaced by a transparent token. If the most active users correlate highly with those most willing to pay, then introducing Session Monetization with a transparent token could either significantly inconvenience these active users or substantially elevate the risk of deanonymization for core users. These core users, who most require privacy, are comparable to the hubs or critical nodes within the network. Making privacy transactions optional would be akin to eliminating mandatory onion routing from the Session network. Should the privacy of these core users be breached, the fallout could extend to their contacts, potentially compromising the privacy of a significant proportion of the network. For further details, see Privacy Implications of Replacing the Oxen Privacy Coin in the ONS Registration Process.
Appendix: Synthesis Formula of MAU from OPTF
Source: https://t.me/Oxen_Community/381121
The text was updated successfully, but these errors were encountered: