care partner alerts #715

ewollesen · 2024-05-06T15:55:54Z

This used to be a series of PRs, but that didn't really work out. They're all collapsed into this one.

Shouldn't be merged until tidepool-org/go-common#71 is merged, then this should have it's go-common bumped.

ewollesen · 2024-07-11T19:11:07Z

To use this in QA, it must be paired with tidepool-org/hydrophone#145 and tidepool-org/go-common#71

toddkazakov

Looks good overall, but the retry mechanism which is implemented here doesn't satisfy the latency requirements. The current implementation is ok for internal usage, but it's not production ready. This could be handled in a separate PR if this makes the development and QA process easier.

auth/store/mongo/device_tokens_repository.go

auth/store/test/device_token_repository.go

data/events/alerts.go

toddkazakov · 2024-07-12T12:56:39Z

data/events/events.go

+	}
+	handler := asyncevents.NewSaramaConsumerGroupHandler(&asyncevents.NTimesRetryingConsumer{
+		Consumer: r.Config.MessageConsumer,
+		Delay:    CappedExponentialBinaryDelay(AlertsEventRetryDelayMaximum),


I don't think this is a suitable retry strategy given the latency requirements for this service. Kafka's consumer group concurrency is limited to the number of partitions of the topic. This number cannot be very high because Kafka's memory consumption grows linearly with the number of partitions. From this follows that the number of partitions is much lower than the number of users we will have and the data of multiple users will end up in the same partition. A failure to evaluate a single user's alerts for one minute as currently set by the CappedExponentialBinaryDelay will introduce at least a minute delay to all of the users sharing the same partition, because messages in a single partition are processed serially.

Alert notifications should be near real-time - up to 10 seconds latency is acceptable. I think the solution proposed in this design document is how this should be handled. Other solutions which satisfy the requirements are welcome.

This will require some more in-depth thought on my part... Will do.

Yeah, I think you're right, let's get this review merged, and I'll work on getting a multiple topic solution set up. Given the flexibility we have now, it shouldn't be too bad.

I have the multi-tier retry in the eric-alerts-multi-topic-retry branch.

This should be implemented in this branch now.

data/events/events.go

data/service/service/standard.go

ewollesen · 2024-10-07T19:28:03Z

@toddkazakov I just removed two config env vars. I believe we talked about that before, but it slipped my mind until I was reviewing the helm chart changes today, where they came up again.

So the re-review here is just around the config parsing, in the most recent commit of the PR, nothing else is changed. [UPDATE] this comment is outdated.

These won't be changing at runtime, so there's no need to complicate the initialization by making these configurable. The topic's prefix is configurable, and that's the part that will change from environment to environment at runtime. BACK-2554

A rebase has picked up work performed by Darin, which removes the need for this token injection. \o/ Yay!

These tests, and the functionality they cover were moved into the alerts/client.go in a previous commit.

BACK-2449

BACK-2499

- UsersWithoutCommunication endpoint added to data service - UsersWithoutCommunication endpoint added to alerts client - implementing no communication alerts via the task service - evaluation of alerts conditions re-worked - The new system recognizes that some alerts are generated by events (so-called "Data Alerts") while others are polled (no communication). - The new evaluator lives in the alerts package (was data/events) - implemented tracking of sent notifications - Recording repo is implemented to record/index the time of the last received data from a user BACK-2558

I've identified that there are some big changes that will need to happen in order to manage marking things sent and resolved. Those will come in a future commit. BACK-2559

The above turned out to be a lot more complicated than I had imagined they'd be. BACK-2559

ewollesen · 2025-01-28T15:27:05Z

Ready for review once again.

As requested in code review. tidepool-org/terraform-modules#72 (review) BACK-2559

BACK-2559 BACK-2499

toddkazakov · 2025-02-05T10:13:27Z

data/store/mongo/mongo.go

@@ -66,3 +71,13 @@ func (s *Store) NewAlertsRepository() alerts.Repository {
 	r := alertsRepo(*s.Store.GetRepository("alerts"))
 	return &r
 }
+
+func (s *Store) NewRecorderRepository() alerts.RecordsRepository {
+	r := recorderRepo(*s.Store.GetRepository("records"))


It's not obvious what is meant by record. Is there a better name that we can use?

toddkazakov · 2025-02-05T10:41:56Z

alerts/tasks.go

+	now := time.Now()
+	if nextDesiredRun.Before(now) {
+		r.logger.Info("care partner is bumping nextDesiredRun")
+		// nextDesiredRun, when added to time.Now in tsk.RepeatAvailableAfter, must


I suggest changing the code of the queue to not fail the task if available time is in the past. It looks like a safeguard to prevent starvation in case we run tasks in an endless loop in case available time is set incorrectly, but now we have a good reason to remove this code. This way we can remove the workaround you came up with which looks fragile and will terminate the task in case the task goes into failed state.

@darinkrauss any thoughts or advice on this before I spend too much time working on it?

@ewollesen I think that the change you'd need to make is, in task/queue/queue.go:

func (q *queue) computeState(tsk *task.Task) { switch tsk.State { case task.TaskStatePending: if tsk.AvailableTime == nil || time.Now().After(*tsk.AvailableTime) { tsk.AppendError(errors.New("pending task requires future available time")) tsk.SetFailed() }

to:

func (q *queue) computeState(tsk *task.Task) { switch tsk.State { case task.TaskStatePending: if tsk.AvailableTime == nil { tsk.AvailableTime = time.Now() } else if time.Now().After(*tsk.AvailableTime) { tsk.AppendError(errors.New("pending task requires future available time")) tsk.SetFailed() }

I think it is okay, but there would need to be a deeper look into Dexcom task to see if it would be an issue. I don't think so, but 🤷

@darinkrauss @toddkazakov : I've made the change, and added a method to make use of it.

I don't see any problems with the dexcom task, but you'd be the expert there for sure Darin. As far as I could see, the dexcom task always either have an available time, or are in an failed state (and so available time doesn't come into play).

toddkazakov · 2025-02-05T10:47:28Z

task/carepartner.go

+
+const CarePartnerType = "org.tidepool.carepartner"
+
+func NewCarePartnerTaskCreate() *TaskCreate {


You should move this to the alerts package.

toddkazakov · 2025-02-05T10:47:56Z

task/carepartner_test.go

+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("NewCarePartnerTaskCreate", func() {


You should move this to the alerts package.

toddkazakov · 2025-02-05T10:53:51Z

data/service/service/standard.go

+		},
+	}
+
+	retryDelays := []time.Duration{0, 1 * time.Second}


Retry delays should be configurable with an env variable in case we want to adjust them, instead of having to modify the code.

Done.

See also tidepool-org/development#314

toddkazakov · 2025-02-05T11:19:10Z

data/store/mongo/mongo_recorder.go

+	}
+}
+
+func (d *recorderRepo) UsersWithoutCommunication(ctx context.Context) ([]alerts.LastCommunication, error) {


Doesn't return "users", consider renaming this.

toddkazakov · 2025-02-05T11:19:38Z

data/store/mongo/mongo_recorder.go

+	structuredmongo "github.com/tidepool-org/platform/store/structured/mongo"
+)
+
+// recorderRepo implements RecorderRepository, writing data to a MongoDB collection.


Rename this to something more descriptive

toddkazakov · 2025-02-05T11:21:07Z

alerts/config.go

+	ctx = log.NewContextWithLogger(ctx, lgr)
+	nc := c.Alerts.NoCommunication.Evaluate(ctx, last)
+	needsUpsert := c.Activity.NoCommunication.Update(nc.OutOfRange)
+	// TODO check re-eval? I don't think so


Open a ticket if this needs to be addressed and remove the TODO comment

toddkazakov · 2025-02-05T11:24:56Z

alerts/config.go

 	"time"

 	"github.com/tidepool-org/platform/data"
-	"github.com/tidepool-org/platform/data/blood/glucose"
+	nontypesglucose "github.com/tidepool-org/platform/data/blood/glucose"


Rename this (and other occurrences) to dataBloodGlucose for consistency. I don't think nontypesglucose has any benefit over the existing alias used throughout the repository.

toddkazakov · 2025-02-05T11:28:05Z

alerts/config.go

+}
+
+// RecordsRepository encapsulates queries of the records collection for use with alerts.
+type RecordsRepository interface {


I think it's important to capture what type of "communications" this repository stores and that those are used solely for alerting purposes. This applies to the struct, interface and collection. Unfortunately, I can't come up with a good suggestion myself.

I've taken a stab at a better name. Let me know what you think.

This maintains consistency. BACK-2499 BACK-2559

Renames: Recorder -> LastCommunicationsRecorder UsersWithoutCommunication -> OverdueCommunications BACK-2499 BACK-2559

Explicit > Implicit BACK-2499 BACK-2559

BACK-2499 BACK-2559

Requested in code review. #715 (comment) BACK-2499 BACK-2559

BACK-2499 BACK-2559

BACK-2559

Previously, when completing a task, an available time of nil would cause the task to be marked as failed. Now, when a task completes and has available time of nil, time.Now() is substituted, which should cause the task to be run again ASAP. This supports the care partner no communication check, which wants to run 1x/second, but as that's not available with the task service (the smallest interval is 5 seconds), setting the value to nil will run the task on each task service iteration. BACK-2559

darinkrauss · 2025-02-13T19:47:37Z

task/queue/queue.go

+		if tsk.AvailableTime == nil {
+			tsk.AvailableTime = pointer.FromAny(time.Now())
+		} else if time.Now().After(*tsk.AvailableTime) {
 			tsk.AppendError(errors.New("pending task requires future available time"))
 			tsk.SetFailed()
 		}


Actually, after thinking a bit more on this, let's change this to:

Suggested change

if tsk.AvailableTime == nil {

tsk.AvailableTime = pointer.FromAny(time.Now())

} else if time.Now().After(*tsk.AvailableTime) {

tsk.AppendError(errors.New("pending task requires future available time"))

tsk.SetFailed()

}

if now := time.Now(); tsk.AvailableTime == nil || tsk.AvailableTime.Before(now) {

tsk.AvailableTime = pointer.FromAny(now)

}

This should fully resolve any edge cases (due to timing, etc.) plus allow for nil to just mean "ASAP".

darinkrauss · 2025-02-13T19:52:06Z

data/events/alerts.go

@@ -0,0 +1,233 @@
+package events


Is this part of the formal (public) Tidepool data model or is it just an internal-only (never to be exposed to the user)? If public, then I'd like to review and comment, but it'll need to wait a bit until I have more time.

darinkrauss · 2025-02-13T19:53:49Z

task/service/service/service.go

+
+	platformConfig := platform.NewConfig()
+	platformConfig.UserAgent = s.UserAgent()
+	reporter := s.ConfigReporter().WithScopes("data", "client")


Why is this using "data" client, shouldn't this be "alert(s)" client?

darinkrauss · 2025-02-13T19:56:40Z

task/service/service/service.go

+func (s *Service) initializePusher() error {
+	var err error
+
+	apns2Config := &struct {


Typically we define the config struct alongside the object is is configuring (APNSPusher, in this case). It allows greatly isolation of knowledge. Also, my personal preference is that we keep using ConfigReporter rather than adding in yet another configuration mechanism.

I would like us to move away from config reporter. With many seemingly similar components which use generic names (services, clients, service clients, standard clients, etc) it's absolutely impossible to determine where an env variable is used and how the configuration is loaded.

I agree, that makes complete sense to me. I agree that ConfigReporter isn't completely obvious and moving to a more direct mechanism for configuration is a good idea.

I think, however, we should have a specific, agreed-upon plan for doing so. I'd rather not have multiple alternatives to ConfigReporter in the code, but one agreed upon solution going forward. Plus, I think we may be able to update the various configs with specific struct fields and apply a surgical change to ConfigReporter that would prevent a large change to remove ConfigReporter (at least in the short-term).

ewollesen requested a review from toddkazakov May 6, 2024 15:55

ewollesen mentioned this pull request May 6, 2024

lift Repeat out of the base alert config #716

Closed

ewollesen removed the request for review from toddkazakov May 8, 2024 19:59

ewollesen force-pushed the eric-cpa-alerts branch from 374bae2 to 2deda23 Compare June 6, 2024 17:52

ewollesen force-pushed the eric-cpa-alerts branch 2 times, most recently from 8549c33 to 8367902 Compare June 24, 2024 19:09

ewollesen force-pushed the eric-cpa-alerts branch from 8367902 to 2ea9686 Compare July 9, 2024 22:43

ewollesen changed the title ~~adds List and Get methods to alerts client~~ minimal implementation of care partner alerts Jul 9, 2024

ewollesen requested a review from toddkazakov July 9, 2024 22:50

ewollesen force-pushed the eric-cpa-alerts branch from 2ea9686 to 7246848 Compare July 10, 2024 17:31

ewollesen force-pushed the eric-cpa-alerts branch 2 times, most recently from c50d589 to 986106b Compare July 11, 2024 22:55

ewollesen mentioned this pull request Jul 11, 2024

add evaluation of not looping alerts #751

Closed

toddkazakov reviewed Jul 12, 2024

View reviewed changes

ewollesen force-pushed the eric-cpa-alerts branch 2 times, most recently from c08e1fc to 967c617 Compare July 19, 2024 16:02

ewollesen requested a review from toddkazakov July 29, 2024 22:41

ewollesen force-pushed the eric-cpa-alerts branch from 967c617 to cd42b19 Compare July 29, 2024 22:44

ewollesen force-pushed the eric-cpa-alerts branch 2 times, most recently from 9432468 to eaa652e Compare September 17, 2024 20:07

toddkazakov previously approved these changes Sep 18, 2024

View reviewed changes

ewollesen dismissed toddkazakov’s stale review via ee5da4a October 2, 2024 17:56

ewollesen force-pushed the eric-cpa-alerts branch from d6449a1 to ee5da4a Compare October 2, 2024 17:56

toddkazakov previously approved these changes Oct 5, 2024

View reviewed changes

ewollesen dismissed toddkazakov’s stale review via 8ae1dc8 October 7, 2024 19:25

ewollesen requested a review from toddkazakov October 7, 2024 19:26

ewollesen force-pushed the eric-cpa-alerts branch from 8ae1dc8 to 28fdf06 Compare October 31, 2024 15:40

ewollesen force-pushed the eric-cpa-alerts branch from 28fdf06 to b9767dc Compare December 11, 2024 19:54

ewollesen added 8 commits January 28, 2025 08:22

there's no longer a need to inject server session tokens

3846520

A rebase has picked up work performed by Darin, which removes the need for this token injection. \o/ Yay!

removes out-of-date tests

38d1c85

These tests, and the functionality they cover were moved into the alerts/client.go in a previous commit.

improve test coverage

19bdcda

BACK-2449

add data set id to alerts Evaluation, improve test coverage

a2be12d

BACK-2499

evaluate not looping conditions part 1

7a81670

I've identified that there are some big changes that will need to happen in order to manage marking things sent and resolved. Those will come in a future commit. BACK-2559

re-working to handle alert resolution and sent tracking

9562b32

The above turned out to be a lot more complicated than I had imagined they'd be. BACK-2559

ewollesen force-pushed the eric-cpa-alerts branch from 2e91daf to 9562b32 Compare January 28, 2025 15:25

ewollesen added 2 commits February 4, 2025 14:57

reduce kafka topics for care partner alerts outside of production

239f89c

As requested in code review. tidepool-org/terraform-modules#72 (review) BACK-2559

bump go-common to get kafka CDC updates for CPA

f49218a

BACK-2559 BACK-2499

ewollesen force-pushed the eric-cpa-alerts branch from 099bf63 to f49218a Compare February 4, 2025 22:54

toddkazakov requested changes Feb 5, 2025

View reviewed changes

ewollesen added 9 commits February 10, 2025 16:51

rename nontypesglucose -> dataBloodGlucose

bfe48fb

This maintains consistency. BACK-2499 BACK-2559

renames Recorder & UsersWithoutCommunications

e6b2cea

Renames: Recorder -> LastCommunicationsRecorder UsersWithoutCommunication -> OverdueCommunications BACK-2499 BACK-2559

pass a log.Logger to EvaluateNoCommunication

7c3f41d

Explicit > Implicit BACK-2499 BACK-2559

remove un-needed comment

55b9150

BACK-2499 BACK-2559

move care partner task definition to alerts package

2aa72c5

BACK-2499 BACK-2559

make GetRunnerDeadline() use a multiple of GetRunnerDurationMaximum()

1140500

Requested in code review. #715 (comment) BACK-2499 BACK-2559

replace magic number with named constant

0586c46

BACK-2499 BACK-2559

adds environment-based config for alerts retry delays

8f3d3ab

BACK-2559

ewollesen force-pushed the eric-cpa-alerts branch from b5dd269 to 0da82d6 Compare February 12, 2025 23:23

ewollesen requested review from darinkrauss and toddkazakov February 12, 2025 23:25

darinkrauss reviewed Feb 13, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

care partner alerts #715

care partner alerts #715

ewollesen commented May 6, 2024 •

edited

Loading

ewollesen commented Jul 11, 2024 •

edited

Loading

toddkazakov left a comment

toddkazakov Jul 12, 2024

ewollesen Jul 12, 2024

ewollesen Jul 19, 2024

ewollesen Jul 29, 2024

ewollesen Dec 11, 2024

ewollesen commented Oct 7, 2024 •

edited

Loading

ewollesen commented Jan 28, 2025

toddkazakov Feb 5, 2025

toddkazakov Feb 5, 2025

ewollesen Feb 12, 2025

darinkrauss Feb 12, 2025

ewollesen Feb 12, 2025 •

edited

Loading

toddkazakov Feb 5, 2025

toddkazakov Feb 5, 2025

toddkazakov Feb 5, 2025

ewollesen Feb 12, 2025

toddkazakov Feb 5, 2025

toddkazakov Feb 5, 2025

toddkazakov Feb 5, 2025

toddkazakov Feb 5, 2025

toddkazakov Feb 5, 2025

ewollesen Feb 12, 2025

darinkrauss Feb 13, 2025

darinkrauss Feb 13, 2025

darinkrauss Feb 13, 2025

darinkrauss Feb 13, 2025

darinkrauss Feb 13, 2025

toddkazakov Feb 13, 2025

darinkrauss Feb 13, 2025


		const CarePartnerType = "org.tidepool.carepartner"

		func NewCarePartnerTaskCreate() *TaskCreate {

care partner alerts #715

Are you sure you want to change the base?

care partner alerts #715

Conversation

ewollesen commented May 6, 2024 • edited Loading

ewollesen commented Jul 11, 2024 • edited Loading

toddkazakov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ewollesen commented Oct 7, 2024 • edited Loading

ewollesen commented Jan 28, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ewollesen Feb 12, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ewollesen commented May 6, 2024 •

edited

Loading

ewollesen commented Jul 11, 2024 •

edited

Loading

ewollesen commented Oct 7, 2024 •

edited

Loading

ewollesen Feb 12, 2025 •

edited

Loading