Sandersaarond/mysql model metrics #98
Conversation
SandersAaronD
force-pushed
the
sandersaarond/mysql-model-metrics
branch
from
53d8df6
to
e068d1e
Compare
| StackID uint64 `json:"stack_id" gorm:"not null;primaryKey"` | ||
| ProcessID uuid.UUID `json:"process_id" gorm:"type:char(36);not null;primaryKey;foreignKey:ProcessID;references:ID"` // Foreign key | ||
| MetricName string `json:"metric_name" gorm:"size:32;not null;primaryKey"` | ||
| StepName string `json:"step_name" gorm:"size:32;not null;primaryKey"` |
There was a problem hiding this comment.
It's not entirely obvious what StepName is and why we need it. We should add comments explaining the fields.
| // Add a custom hook if necessary for additional logic. | ||
| // Example: AfterCreate hook for custom logic | ||
| func (m *ModelMetrics) AfterCreate(tx *gorm.DB) error { | ||
| // Custom logic after creating a metric entry | ||
| tx.Logger.Info(tx.Statement.Context, "AfterCreate hook called for ModelMetrics") | ||
| return nil | ||
| } |
There was a problem hiding this comment.
I don't think this is being used?
| type testApp struct { | ||
| App | ||
| } | ||
|
|
||
| func (a *testApp) db(ctx context.Context) *gorm.DB { | ||
| return a.App._db | ||
| } | ||
|
|
||
| func setupTestDB(t *testing.T) (*gorm.DB, func()) { | ||
| db, err := gorm.Open(sqlite.Open("file::memory:?cache=shared"), &gorm.Config{}) | ||
| require.NoError(t, err) | ||
|
|
||
| err = db.AutoMigrate(&model.Process{}, &model.ModelMetrics{}) | ||
| require.NoError(t, err) | ||
|
|
||
| return db, func() { | ||
| sqlDB, err := db.DB() | ||
| require.NoError(t, err) | ||
| sqlDB.Close() | ||
| } | ||
| } |
There was a problem hiding this comment.
Can we reuse the code in api_test.go? I think most of this boilerplate is set up already.
| ) | ||
|
|
||
| type ModelMetrics struct { | ||
| StackID uint64 `json:"stack_id" gorm:"not null;primaryKey"` |
There was a problem hiding this comment.
Can we use TenantID as that's the consistent key we are using in other structs? Or can we update the other structs to use StackID as a uint64?
| } | ||
|
|
||
| // Iterate over the metrics and build the series data | ||
| var response GetModelMetricsResponse |
There was a problem hiding this comment.
I think there's some indentation mismatch in this file from this point on
| // Incoming format is an array of these | ||
| type ModelMetricsSeries struct { | ||
| MetricName string `json:"metric_name"` | ||
| StepName string `json:"step_name"` | ||
| Points []struct { | ||
| Step uint32 `json:"step"` | ||
| Value json.Number `json:"value"` | ||
| } `json:"points"` | ||
| } |
There was a problem hiding this comment.
Why are we storing metrics as individual rows (one row each for a step and value) in mySQL? Why not store an array like in this struct? Is it not performant? Curious if there was some benchmarking done to choose the former.
There was a problem hiding this comment.
Agreed, it would also limit us from compressing the points in the future which could end up with a lot more disk and network usage. Do we ever not pull back all of the points at once?
| MetricValue string `json:"metric_value" gorm:"size:64;not null"` | ||
|
|
||
| Process Process `gorm:"foreignKey:ProcessID;references:ID"` // Relationship definition | ||
| } |
There was a problem hiding this comment.
should we also track timestamp for each step? i feel that might be helpful in tracking time between training runs, performance tracking, etc.
WIP
Meant to transition writing and reading model metrics from loki to mysql, currently has a writer implemented and the reader function is there but not attached to an endpoint.
TODO for this:
Unclear if this can be done as part of this PR, but: I have a dangling concern about authentication, where it isn't clear to me if tenantID will always be populated correctly in production, since our local setup just defaults to 0, making this hard to test locally. Either we need to find a way to test this locally or we won't know about that until we deploy it.