From f441a9d914e9222c4a5054cd960969b090d818db Mon Sep 17 00:00:00 2001 From: Juraj Majerik Date: Mon, 16 Dec 2024 15:04:04 +0100 Subject: [PATCH 1/4] update --- contents/teams/experiments/objectives.mdx | 55 +++++++++++++++-------- 1 file changed, 36 insertions(+), 19 deletions(-) diff --git a/contents/teams/experiments/objectives.mdx b/contents/teams/experiments/objectives.mdx index 1a4167f80e62..740e23784126 100644 --- a/contents/teams/experiments/objectives.mdx +++ b/contents/teams/experiments/objectives.mdx @@ -1,19 +1,36 @@ -### Q4 2024 Objectives - -This quarter, we're focusing on more feature improvements to help users create comprehensive and easy-to-analyze experiments. These improvements will make our product more mature and complete. - -### Objective 1: Finish HogQL rewrite - -We will migrate Experiments to HogQL, making result calculations more reliable and performant. This will also enable the addition of new features listed below. - -### Objective 2: Multiple experiment goals supported + visualized - -We will add support for multiple goal metrics in a single experiment, allowing them to be visualized together and making it easier to interpret results across all metrics simultaneously. - -### Objective 3: Reusable experiment metrics - -We will add the ability to create metric sets that users can save and reuse. This will reduce friction, improve maintainability, and decrease the likelihood of errors. - -### Objective 4: Review and adjust methodology with a statistician - -We will review our current methodology to ensure it is both accurate and easy for our users to understand. +### Q1 2025 Objectives + +This quarter, we will bring new feature improvements to make the Experiments more advanced. We'll also focus on improving quality across different areas - data, documentation, and codebase - to help us move faster and deliver a better product. + +#### Objective 1: New features +- [Timeseries chart for deltas and credible intervals](https://github.com/PostHog/posthog/issues/26931) @jurajmajerik +- [Winsorization](https://github.com/PostHog/posthog/issues/26060) to exclude extreme values from analyses. @andehen +- [New data collection calculator](https://github.com/PostHog/posthog/issues/26933) @andehen +- [Filtering for experiment results](https://github.com/PostHog/posthog/issues/26934) @danielbachhuber + +#### Objective 2: Data & statistics +- Complete the [migration to updated statistical methods](https://github.com/PostHog/posthog/issues/26713). +- Document each statistical method in detail. +- Write a practical "manual" chapter for every metric type. +- Address neglected cases, such as "average count per user." +- Improve diagnostics to provide more detailed information, like flagging missing exposure data. @andehen + +#### Objective 3: Data Warehouse integration @danielbachhuber +- Ensure a great experience for our pilot users. +- Add support for funnel experiments. +- Graduate the integration out of beta. + +#### Objective 4: UX & UI +- Polish the reusable metrics UI. @jurajmajerik +- Add new summary banners to support multiple metrics. +- Improve product analytics insights when transitioning from experiments. +- [Polish no-code experiments UI](https://github.com/PostHog/posthog/issues/26936) +- Add [in-product tooltips](https://github.com/PostHog/posthog/issues/26937). + +#### Objective 5: Codebase quality +- Improve test coverage for query runners and add missing legacy code test cases. +- Add E2E for the basic flows once the multi-metrics UI stabilizes. +- Avoid global updates from modals that require page reloads on Cancel. +- Refactor experiment logic to remove dependencies on logics, improve reusability, and delete dead code. +- Set up monitoring for query timeouts. +- Audit Sentry to ensure we’re tracking the most relevant issues. From 6f7af89c99383401666e37b9145cf1741e9368d0 Mon Sep 17 00:00:00 2001 From: Juraj Majerik Date: Mon, 16 Dec 2024 15:07:49 +0100 Subject: [PATCH 2/4] typo --- contents/teams/experiments/objectives.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/contents/teams/experiments/objectives.mdx b/contents/teams/experiments/objectives.mdx index 740e23784126..731c622e6bc3 100644 --- a/contents/teams/experiments/objectives.mdx +++ b/contents/teams/experiments/objectives.mdx @@ -1,6 +1,6 @@ ### Q1 2025 Objectives -This quarter, we will bring new feature improvements to make the Experiments more advanced. We'll also focus on improving quality across different areas - data, documentation, and codebase - to help us move faster and deliver a better product. +This quarter, we will bring new feature improvements to make Experiments more advanced. We'll also focus on improving quality across different areas - data, documentation, and codebase - to help us move faster and deliver a better product. #### Objective 1: New features - [Timeseries chart for deltas and credible intervals](https://github.com/PostHog/posthog/issues/26931) @jurajmajerik From f30d7f5446b4c7b56bf10553c6e77757ce9849ef Mon Sep 17 00:00:00 2001 From: Juraj Majerik Date: Mon, 16 Dec 2024 15:10:12 +0100 Subject: [PATCH 3/4] typo --- contents/teams/experiments/objectives.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/contents/teams/experiments/objectives.mdx b/contents/teams/experiments/objectives.mdx index 731c622e6bc3..ee176441d291 100644 --- a/contents/teams/experiments/objectives.mdx +++ b/contents/teams/experiments/objectives.mdx @@ -31,6 +31,6 @@ This quarter, we will bring new feature improvements to make Experiments more ad - Improve test coverage for query runners and add missing legacy code test cases. - Add E2E for the basic flows once the multi-metrics UI stabilizes. - Avoid global updates from modals that require page reloads on Cancel. -- Refactor experiment logic to remove dependencies on logics, improve reusability, and delete dead code. +- Refactor experiment logic to remove dependencies on other logics, improve reusability, and delete dead code. - Set up monitoring for query timeouts. - Audit Sentry to ensure we’re tracking the most relevant issues. From 3915550c636b985c59808103d449c4723eedd59b Mon Sep 17 00:00:00 2001 From: Juraj Majerik Date: Thu, 19 Dec 2024 18:07:44 +0100 Subject: [PATCH 4/4] update --- contents/teams/experiments/objectives.mdx | 33 ++++++++++++++++------- 1 file changed, 23 insertions(+), 10 deletions(-) diff --git a/contents/teams/experiments/objectives.mdx b/contents/teams/experiments/objectives.mdx index ee176441d291..eaedf51c66c4 100644 --- a/contents/teams/experiments/objectives.mdx +++ b/contents/teams/experiments/objectives.mdx @@ -1,36 +1,49 @@ ### Q1 2025 Objectives -This quarter, we will bring new feature improvements to make Experiments more advanced. We'll also focus on improving quality across different areas - data, documentation, and codebase - to help us move faster and deliver a better product. +Experiments at PostHog are growing up. Our product is simple yet functional, but there are gaps compared to our competitors. This quarter, we're balancing quality improvements with building new features to make PostHog Experiments a great choice for teams of any size and maturity. To attract mature users and retain our growing base, we must close those gaps while improving the overall product experience. -#### Objective 1: New features +Last quarter, we scaled the team from one to three engineers and shipped a lot of features, but we've accumulated technical debt and seen an increase in support tickets. This quarter, we aim to fix bugs, improve documentation, and solidify the product while keeping feature work manageable to have a stable foundation for future growth. + +We have five core objectives for Q1: +1. New features: Keep closing feature gaps to attract advanced users. +2. Data & statistics: Finalize our new statistics engine to improve correctness and clarity. +3. Data Warehouse integration: Lay the groundwork for large-scale adoption. +4. UX & UI improvements: Make the interface simple and intuitive. +5. Codebase quality: Clean up technical debt to speed up future development. + +#### Objective 1: New features (@jurajmajerik, @danielbachhuber, @andehen) +These features are consistently requested by our largest and most demanding customers, but will greatly benefit all users. - [Timeseries chart for deltas and credible intervals](https://github.com/PostHog/posthog/issues/26931) @jurajmajerik - [Winsorization](https://github.com/PostHog/posthog/issues/26060) to exclude extreme values from analyses. @andehen - [New data collection calculator](https://github.com/PostHog/posthog/issues/26933) @andehen - [Filtering for experiment results](https://github.com/PostHog/posthog/issues/26934) @danielbachhuber -#### Objective 2: Data & statistics +#### Objective 2: Data & statistics (@andehen) +We’re finalizing our new statistics engine to ensure more accurate and trustworthy experiment results. - Complete the [migration to updated statistical methods](https://github.com/PostHog/posthog/issues/26713). - Document each statistical method in detail. - Write a practical "manual" chapter for every metric type. - Address neglected cases, such as "average count per user." -- Improve diagnostics to provide more detailed information, like flagging missing exposure data. @andehen +- Improve diagnostics to provide more detailed information, like flagging missing exposure data. -#### Objective 3: Data Warehouse integration @danielbachhuber +#### Objective 3: Data Warehouse integration (@danielbachhuber) +Large customers increasingly request this capability, and our major competitors already offer it. This is a must-have for enterprise-level adoption. - Ensure a great experience for our pilot users. - Add support for funnel experiments. - Graduate the integration out of beta. -#### Objective 4: UX & UI -- Polish the reusable metrics UI. @jurajmajerik -- Add new summary banners to support multiple metrics. +#### Objective 4: UX & UI improvements (@jurajmajerik) +A great user experience is one of the few areas where we can already outcompete our competitors. +- Polish the reusable metrics UI. +- Add a new verbal summary to support multiple metrics. - Improve product analytics insights when transitioning from experiments. - [Polish no-code experiments UI](https://github.com/PostHog/posthog/issues/26936) - Add [in-product tooltips](https://github.com/PostHog/posthog/issues/26937). -#### Objective 5: Codebase quality +#### Objective 5: Codebase quality (@jurajmajerik) +We've accumulated technical debt over the last year and now need to clean up the codebase to allow us to move faster. - Improve test coverage for query runners and add missing legacy code test cases. - Add E2E for the basic flows once the multi-metrics UI stabilizes. -- Avoid global updates from modals that require page reloads on Cancel. - Refactor experiment logic to remove dependencies on other logics, improve reusability, and delete dead code. - Set up monitoring for query timeouts. - Audit Sentry to ensure we’re tracking the most relevant issues.