Skip to content

feat: add utility functions for scheduler #66

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
- [Key-Value Pairs](libs/pairs.md)
- [Readiness Checks](libs/readiness.md)
- [Kubernetes Resource Management](libs/resource.md)
- [Retrying k8s Operations](libs/retry.md)
- [Kubernetes Resource Status Updating](libs/status.md)
- [Testing](libs/testing.md)
- [Thread Management](libs/threads.md)
Expand Down
26 changes: 26 additions & 0 deletions docs/libs/retry.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# Retrying k8s Operations

The `pkg/retry` package contains a `Client` that wraps a `client.Client` while implementing the interface itself and retries any failed (= the returned error is not `nil`) operation.
Methods that don't return an error are simply forwarded to the internal client.

In addition to the `client.Client` interface's methods, the `retry.Client` also has `CreateOrUpdate` and `CreateOrPatch` methods, which use the corresponding controller-runtime implementations internally.

The default retry parameters are:
- retry every 100 milliseconds
- don't increase retry interval
- no maximum number of attempts
- timeout after 1 second

The `retry.Client` struct has builder-style methods to configure the parameters:
```golang
retryingClient := retry.NewRetryingClient(myClient).
WithTimeout(10 * time.Second). // try for at max 10 seconds
WithInterval(500 * time.Millisecond). // try every 500 milliseconds, but ...
WithBackoffMultiplier(2.0) // ... double the interval after each retry
```

For convenience, the `clusters.Cluster` type can return a retrying client for its internal client:
```golang
// cluster is of type *clusters.Cluster
err := cluster.Retry().WithMaxAttempts(3).Get(...)
```
7 changes: 7 additions & 0 deletions pkg/clusters/cluster.go
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import (
"sigs.k8s.io/controller-runtime/pkg/cluster"

"github.com/openmcp-project/controller-utils/pkg/controller"
"github.com/openmcp-project/controller-utils/pkg/retry"
)

type Cluster struct {
Expand Down Expand Up @@ -238,6 +239,12 @@ func (c *Cluster) APIServerEndpoint() string {
return c.restCfg.Host
}

// Retry returns a retrying client for the cluster.
// Returns nil if the client has not been initialized.
func (c *Cluster) Retry() *retry.Client {
return retry.NewRetryingClient(c.Client())
}

/////////////////
// Serializing //
/////////////////
Expand Down
38 changes: 37 additions & 1 deletion pkg/collections/maps/utils.go
Original file line number Diff line number Diff line change
@@ -1,6 +1,11 @@
package maps

import "github.com/openmcp-project/controller-utils/pkg/collections/filters"
import (
"k8s.io/utils/ptr"

"github.com/openmcp-project/controller-utils/pkg/collections/filters"
"github.com/openmcp-project/controller-utils/pkg/pairs"
)

// Filter filters a map by applying a filter function to each key-value pair.
// Only the entries for which the filter function returns true are kept in the copy.
Expand Down Expand Up @@ -50,3 +55,34 @@ func Intersect[K comparable, V any](source map[K]V, maps ...map[K]V) map[K]V {

return res
}

// MapKeys returns a slice of all keys in the map.
// The order is unspecified.
// The keys are not deep-copied, so changes to them could affect the original map.
func MapKeys[K comparable, V any](m map[K]V) []K {
keys := make([]K, 0, len(m))
for k := range m {
keys = append(keys, k)
}
return keys
}
Comment on lines +59 to +68
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we instead use the Keys function from the standard package here as well: https://pkg.go.dev/maps#Keys? Or does this not work, with what you have planned?


// MapValues returns a slice of all values in the map.
// The order is unspecified.
// The values are not deep-copied, so changes to them could affect the original map.
func MapValues[K comparable, V any](m map[K]V) []V {
values := make([]V, 0, len(m))
for _, v := range m {
values = append(values, v)
}
return values
}
Comment on lines +70 to +79
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same goes here with https://pkg.go.dev/maps#Values as well. What do you think about that?


// GetAny returns an arbitrary key-value pair from the map as a pointer to a pairs.Pair.
// If the map is empty, it returns nil.
func GetAny[K comparable, V any](m map[K]V) *pairs.Pair[K, V] {
for k, v := range m {
return ptr.To(pairs.New(k, v))
}
return nil
}
54 changes: 53 additions & 1 deletion pkg/collections/maps/utils_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ import (
"github.com/openmcp-project/controller-utils/pkg/collections/maps"
)

var _ = Describe("LinkedIterator Tests", func() {
var _ = Describe("Map Utils Tests", func() {

Context("Merge", func() {

Expand Down Expand Up @@ -60,4 +60,56 @@ var _ = Describe("LinkedIterator Tests", func() {

})

Context("MapKeys", func() {

It("should return all keys in the map", func() {
m1 := map[string]string{"foo": "bar", "bar": "baz", "foobar": "foobaz"}
keys := maps.MapKeys(m1)
Expect(keys).To(ConsistOf("foo", "bar", "foobar"))
Expect(len(keys)).To(Equal(3))
})

It("should return an empty slice for an empty or nil map", func() {
var nilMap map[string]string
Expect(maps.MapKeys(nilMap)).To(BeEmpty())
Expect(maps.MapKeys(map[string]string{})).To(BeEmpty())
})

})

Context("MapValues", func() {

It("should return all values in the map", func() {
m1 := map[string]string{"foo": "bar", "bar": "baz", "foobar": "foobaz"}
values := maps.MapValues(m1)
Expect(values).To(ConsistOf("bar", "baz", "foobaz"))
Expect(len(values)).To(Equal(3))
})

It("should return an empty slice for an empty or nil map", func() {
var nilMap map[string]string
Expect(maps.MapValues(nilMap)).To(BeEmpty())
Expect(maps.MapValues(map[string]string{})).To(BeEmpty())
})

})

Context("GetAny", func() {

It("should return a key-value pair from the map", func() {
m1 := map[string]string{"foo": "bar", "bar": "baz", "foobar": "foobaz"}
pair := maps.GetAny(m1)
Expect(pair).ToNot(BeNil())
Expect(pair.Key).To(BeElementOf("foo", "bar", "foobar"))
Expect(m1[pair.Key]).To(Equal(pair.Value))
})

It("should return nil for an empty or nil map", func() {
var nilMap map[string]string
Expect(maps.GetAny(nilMap)).To(BeNil())
Expect(maps.GetAny(map[string]string{})).To(BeNil())
})

})

})
78 changes: 78 additions & 0 deletions pkg/collections/utils.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
package collections

// ProjectSlice takes a slice and a projection function and applies this function to each element of the slice.
// It returns a new slice containing the results of the projection.
// The original slice is not modified.
// If the projection function is nil, it returns nil.
func ProjectSlice[X any, Y any](src []X, project func(X) Y) []Y {
if project == nil {
return nil
}
res := make([]Y, len(src))
for i, src := range src {
res[i] = project(src)
}
return res
}

// ProjectMapToSlice takes a map and a projection function and applies this function to each key-value pair in the map.
// It returns a new slice containing the results of the projection.
// The original map is not modified.
// If the projection function is nil, it returns nil.
func ProjectMapToSlice[K comparable, V any, R any](src map[K]V, project func(K, V) R) []R {
if project == nil {
return nil
}
res := make([]R, 0, len(src))
for k, v := range src {
res = append(res, project(k, v))
}
return res
}

// ProjectMapToMap takes a map and a projection function and applies this function to each key-value pair in the map.
// It returns a new map containing the results of the projection.
// The original map is not modified.
// Note that the resulting map may be smaller if the projection function does not guarantee unique keys.
// If the projection function is nil, it returns nil.
func ProjectMapToMap[K1 comparable, V1 any, K2 comparable, V2 any](src map[K1]V1, project func(K1, V1) (K2, V2)) map[K2]V2 {
if project == nil {
return nil
}
res := make(map[K2]V2, len(src))
for k, v := range src {
newK, newV := project(k, v)
res[newK] = newV
}
return res
}

// AggregateSlice takes a slice, an aggregation function and an initial value.
// It applies the aggregation function to each element of the slice, also passing in the current result.
// For the first element, it uses the initial value as the current result.
// Returns initial if the aggregation function is nil.
func AggregateSlice[X any, Y any](src []X, agg func(X, Y) Y, initial Y) Y {
if agg == nil {
return initial
}
res := initial
for _, x := range src {
res = agg(x, res)
}
return res
}

// AggregateMap takes a map, an aggregation function and an initial value.
// It applies the aggregation function to each key-value pair in the map, also passing in the current result.
// For the first key-value pair, it uses the initial value as the current result.
// Returns initial if the aggregation function is nil.
func AggregateMap[K comparable, V any, R any](src map[K]V, agg func(K, V, R) R, initial R) R {
if agg == nil {
return initial
}
res := initial
for k, v := range src {
res = agg(k, v, res)
}
return res
}
93 changes: 93 additions & 0 deletions pkg/collections/utils_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
package collections_test

import (
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"

"github.com/openmcp-project/controller-utils/pkg/collections"
)

var _ = Describe("Utils Tests", func() {

Context("ProjectSlice", func() {

projectFunc := func(i int) int {
return i * 2
}

It("should use the projection function on each element of the slice", func() {
src := []int{1, 2, 3, 4}
projected := collections.ProjectSlice(src, projectFunc)
Expect(projected).To(Equal([]int{2, 4, 6, 8}))
Expect(src).To(Equal([]int{1, 2, 3, 4}), "original slice should not be modified")
})

It("should return an empty slice for an empty or nil input slice", func() {
Expect(collections.ProjectSlice(nil, projectFunc)).To(BeEmpty())
Expect(collections.ProjectSlice([]int{}, projectFunc)).To(BeEmpty())
})

It("should return nil for a nil projection function", func() {
src := []int{1, 2, 3, 4}
projected := collections.ProjectSlice[int, int](src, nil)
Expect(projected).To(BeNil())
Expect(src).To(Equal([]int{1, 2, 3, 4}), "original slice should not be modified")
})

})

Context("ProjectMapToSlice", func() {

projectFunc := func(k string, v string) string {
return k + ":" + v
}

It("should use the projection function on each key-value pair of the map", func() {
src := map[string]string{"a": "1", "b": "2", "c": "3"}
projected := collections.ProjectMapToSlice(src, projectFunc)
Expect(projected).To(ConsistOf("a:1", "b:2", "c:3"))
Expect(src).To(Equal(map[string]string{"a": "1", "b": "2", "c": "3"}), "original map should not be modified")
})

It("should return an empty slice for an empty or nil input map", func() {
Expect(collections.ProjectMapToSlice(nil, projectFunc)).To(BeEmpty())
Expect(collections.ProjectMapToSlice(map[string]string{}, projectFunc)).To(BeEmpty())
})

It("should return nil for a nil projection function", func() {
src := map[string]string{"a": "1", "b": "2", "c": "3"}
projected := collections.ProjectMapToSlice[string, string, string](src, nil)
Expect(projected).To(BeNil())
Expect(src).To(Equal(map[string]string{"a": "1", "b": "2", "c": "3"}), "original map should not be modified")
})

})

Context("ProjectMapToMap", func() {

projectFunc := func(k string, v string) (string, int) {
return k, len(v)
}

It("should use the projection function on each key-value pair of the map", func() {
src := map[string]string{"a": "1", "b": "22", "c": "333"}
projected := collections.ProjectMapToMap(src, projectFunc)
Expect(projected).To(Equal(map[string]int{"a": 1, "b": 2, "c": 3}))
Expect(src).To(Equal(map[string]string{"a": "1", "b": "22", "c": "333"}), "original map should not be modified")
})

It("should return an empty map for an empty or nil input map", func() {
Expect(collections.ProjectMapToMap(nil, projectFunc)).To(BeEmpty())
Expect(collections.ProjectMapToMap(map[string]string{}, projectFunc)).To(BeEmpty())
})

It("should return nil for a nil projection function", func() {
src := map[string]string{"a": "1", "b": "22", "c": "333"}
projected := collections.ProjectMapToMap[string, string, string, int](src, nil)
Expect(projected).To(BeNil())
Expect(src).To(Equal(map[string]string{"a": "1", "b": "22", "c": "333"}), "original map should not be modified")
})

})

})
26 changes: 26 additions & 0 deletions pkg/controller/utils.go
Original file line number Diff line number Diff line change
Expand Up @@ -52,3 +52,29 @@ func ObjectKey(name string, maybeNamespace ...string) client.ObjectKey {
Name: name,
}
}

// RemoveFinalizersWithPrefix removes finalizers with a given prefix from the object and returns their suffixes.
// If the third argument is true, all finalizers with the given prefix are removed, otherwise only the first one.
// The bool return value indicates whether a finalizer was removed.
// If it is true, the slice return value holds the suffixes of all removed finalizers (will be of length 1 if removeAll is false).
// If it is false, no finalizer with the given prefix was found. The slice return value will be empty in this case.
// The logic is based on the controller-runtime's RemoveFinalizer function.
func RemoveFinalizersWithPrefix(obj client.Object, prefix string, removeAll bool) ([]string, bool) {
fins := obj.GetFinalizers()
length := len(fins)
suffixes := make([]string, 0, length)
found := false

index := 0
for i := range length {
if (removeAll || !found) && strings.HasPrefix(fins[i], prefix) {
suffixes = append(suffixes, strings.TrimPrefix(fins[i], prefix))
found = true
continue
}
fins[index] = fins[i]
index++
}
obj.SetFinalizers(fins[:index])
return suffixes, length != index
}
Loading