Re-imagining Label Reuse

[johnaohara]

In light of #1499 and #1603 and the changes that are planned as a result of these discussion, we should take the time to re-imagine what "re-usable labels" look like.

Re-usable labels should attempt to resolve the issues with the current "Schema" implementation (#1603);

• re-use: reduce scope of changes, so that teams changing label definitions can not not interfere with other tests
• versioning: remove the need to copy label definitions between "Schemas" (or the replacement concept), such that labels and be updated without users having to recreate entire "Schemas". The composable label proposal (Composable Labels #1499) would replace versioning at the label level, and remove the need to have multiple versions of schemas.
• schema: the name "Schema" should be reserved for JSON Validation Scheamas
• uri: at the moment, Horreum uses a uri (or referenced as a urn) to identify a "Schema", but in practice this is a simple unique string used to resolve a "Schema" by name, without validation or enforcement of a particular.

A number of use case present themselves wrt label re-use;

• As a user of Horreum, I would like to be able to re-use previously defined labels, either for previous tests or standard labels for well know tools. Updates to the previously defined labels should be isolated and not affect my currently defined labels, but I should be amde aware of updates to copied labels. As sets of labels evolve, i.e. a newer version of a well used tool is released that changes the output, I should be able to use those label definitions in any tests that use that newer tool. I should not have to define which set of labels required to process a run when I upload a run.

• As a user of Horreum, changes to Label definitions should automatically be reflected in all locations that the label is referenced, without me having to navigate to multiple configuration areas to redefine label usage

• As a team manager, if there labels defined for a test, I should be able to define a set of labels that can be re-used for other tests in my team.

• As a administrator of Horreum, I should be able to define sets of labels that can be used globaly by all teams.

[willr3]

I'm puzzled by some aspects to how we support label reuse (I'm calling them label sets for simplicity) in some scenarios.

Foremost is how does a user indicate where a label set should apply? The current Horreum implementation uses $schema to indicate where a label set (currently called Schema's in production) apply the labels. The PoC had a concept of target_schema which was a tag to indicate labelValues were comparable. We could use that same approach where the target_schema is the name of a set of labels and the user provides a label that creates the json data structure for the set of labels. The down side to this approach is that users must provide a jsonpath that can identify where the appropriate data is located which normally means having some knowledge about the content of the data. The current $schema approach has the advantage of supporting any data that already includes that key.

The other puzzling point is how the user can define a label that derives from labelValues from the label set without adding to the label set. The best idea I've come with is to add the label set name to the label name in the extractor path labelSetName:labelName:$.path.to.data. I'm not sure on the exact syntax. I'm also not sure how we handle the possibility of multiple labels having the same target_schema and then wanting to apply a subsequent label to only one of those. How does the user differentiate the labels that were applied to one instance of the target_schema versus the other?

[johnaohara]

+1 for "Label Sets" atm.

1. So a current benefit of $schema is that it is embeded in the run json payload, but it also comes with some a limitations;
   User data is mutable
   Horreum can not support JSON Arrays as root objects (JSON validation schema of type "array" not supported #677),
   Schemas are only supported in the top 2 levels of a JSON document.

   In order to decouple the $schema from the run data, that mapping needs to be stored somewhere, and is possible to be different between tests. If we move towards a "copy" strategy when a label set is used by a test, a "Label Root" could be defined, that encodes a custom location in an uploaded doc.

2. I think this depends on how LabelSets are implement, whether they are shared (as Schemas are now), or they are copied into a Test and are independent, I think the implemenation would differ. I think we need to explore this some more

I think there is another concept to consider, which is ther "URI" that is used to identify "Schemas" now. the Schema URI is effectively a String named to lookup Schemas, but there is no defined syntax. I was thinking that part of defining "Label Sets", especially if Horreum needs to differentiate between different scopes of shareable labels, we could introduce a URI syntax that defines labels, label sets for tests, e.g;

global:labelSet:Hyperfoil:0.1 (or the labels themselves could be versioned)
global:labelSet:Hyperfoil:0.2

perf-team:labelSet:my_bench:0.1 <- define a version labelSet

or

perf-team:test:my_bench:labels:throughput:0.1 <- define a version Label for a particular Test, labels can have their own version lifecycle

[lampajr]

+1 as well on the "Label Sets" atm (or "Label Group").

I am just thinking loud here but what about this possible design:

• A label is the atomic part here

I was thinking that part of defining "Label Sets", especially if Horreum needs to differentiate between different scopes of shareable labels, we could introduce a URI syntax that defines labels, label sets for tests, e.g;
+1

• I really like the "scoping" approach, such that defined labels can have different scopes/namespaces in order to avoid that other teams can modify other team's labels:

  • Global level (can be used by anyone but only administrators can change them)
  • Team level (can be used by anyone but only team managers can update them / or administrators)
  • Test level (can be used by anyone and any tester working on that team can update them / or managers / or administrators)

• What we called "Label set" is just a group of labels, similarly to what is a Schema at the moment but much more lightweight

Now users can associate to a Test, either individual Labels (building a sort of acyclic graph or pipeline) or a Label set that automatically includes a set of labels to be considered.

The other puzzling point is how the user can define a label that derives from labelValues from the label set without adding to the label set. The best idea I've come with is to add the label set name to the label name in the extractor path labelSetName:labelName:$.path.to.data

TBH I was thinking label sets as just handy set of labels, mainly to avoid users to explicitly associate each label one by one to a Test. Thus, a labelValue can be derived from a label (completely ignoring the fact that it might belong to a label set)

[johnaohara]

"Label set" is just a group of labels, similarly to what is a Schema at the moment but much more lightweight

so a "Label Set" would be just the Labels definition of a current "Schema"

But a crucial difference we need to consider is, are "Label Sets";

• referenced by the tests/runs - this is how schemas are currently implemeneted
• copied into test definitions - this would remove the interference between tests we currently experience with Schemas

each option has is benefits / drawbacks

[lampajr]

I think to answer this question we should ask ourselves "do we want that a change on Label that is referenced by multiple tests will affect all those references? and therefore re-trigger recalcuation and whatver depend on it?"

• If yes, I think that reference would better fit this use case (especially if we consider introducing namespacing/scoping of those Labels)
• If not, maybe a copy can be better.

Either option we choose, I think that providing copy capabilities is something that could be very useful for users but I think it is not strictly related to this topic (it is just an additional funcionality).

[lampajr]

Furthermore, based on how I imagined the design.. Label set are just handy mechanism to simplify the user's worklfow.
I mean, users can directly reference labels one by one without using the label set but with the same result.

[johnaohara]

I was thinking that as part of a Test definition workflow;

The user could select a Label Set to include

Where they could then customise the Labels

Even if the Labels are copied from the Label Set, a reference to the original Label definition could be stored, so that if an update occurs, the User could be presented with an option of how to proceed

[lampajr]

yeah, that is very close to what I was imagining as a possible workflow!

Where they could then customise the Labels

If we choose to reference approach, in this case a new label would be created (copied from the original one)

I still don't know if customizing a label == customize the selected label set.
If yes, in that case we could probably create a copy of the label set, that references the customized (and new) label - at this point the new label set become a Test-specific label set and every additional customization on any other label would be reflacted in that custom label set.

Even if the Labels are copied from the Label Set, a reference to the original Label definition could be stored

+1

BTW I like the look and feel of this workflow 😄

[willr3]

In current Horreum users can export their schemas and tests to json. Does Horreum plan to keep that option for users? The PoC design without label sets supports that option but if we introduce label sets with copy on reference then serialization might impact our design decisions.

The use case I am considering is when a label outputs something that has a known shape and a user wants to use a label set on that output. For example, If a user defines a label "hyperfoil" with extractor "$.hyperfoil" they could want to apply the hyperfoil label set to the output of that label.
Similarly, they may have a label "prometheusMetrics" with an extractor "prometheus[]" that iterates over all the array entries from a previous "prometheus" label. The user could want to apply the prometheus label set to each iteration in "prometheusMetrics". If the upload collected 10 metrics from prometheus then the user would be applying the label set 10 times with the same parent label. Subsequently, they could want to use the output of the labels in the prometheus label set (for example perhaps a label that access the "values" array) in a label that is not part of the label set (perhaps something that gets the avg of the observed metric). We need a way for the user to interact with the labels that are applied by the label set and also be able to specify where the label sets should be applied.
I believe we have the starts of this in the existing PoC with the target_schema tagging that we introduced. That was added so that users to could group label values similar to how they are grouped in production Horreum using datasets. It could also be used to apply a label set to the output of a label. I then think the labels from the referenced label set should be read only from the user's perspective becuase they are part of a known set and not part of the labels created for the test.