Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update dskit and mimir-prometheus #8632

Merged
merged 3 commits into from
Jul 5, 2024
Merged

Update dskit and mimir-prometheus #8632

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c0f1827
Update dskit and mimir-prometheus
pracucci Jul 5, 2024
bf892f0
Added missing files
pracucci Jul 5, 2024
ae7d6dd
Updated CHANGELOG
pracucci Jul 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
* `-ingest-storage.read-consistency`: configures the default read consistency.
* `-ingest-storage.migration.distributor-send-to-ingesters-enabled`: enabled tee-ing writes to classic ingesters and Kafka, used during a live migration to the new ingest storage architecture.
* `-ingester.partition-ring.*`: configures partitions ring backend.
* [FEATURE] Querier: added support for `limitk()` and `limit_ratio()` experimental PromQL functions. Experimental functions are disabled by default, but can be enabled setting `-querier.promql-experimental-functions-enabled=true` in the query-frontend and querier. #8632
* [CHANGE] Distributor: Incoming OTLP requests were previously size-limited by using limit from `-distributor.max-recv-msg-size` option. We have added option `-distributor.max-otlp-request-size` for limiting OTLP requests, with default value of 100 MiB. #8574
* [ENHANCEMENT] Compactor: Add `cortex_compactor_compaction_job_duration_seconds` and `cortex_compactor_compaction_job_blocks` histogram metrics to track duration of individual compaction jobs and number of blocks per job. #8371
* [ENHANCEMENT] Rules: Added per namespace max rules per rule group limit. The maximum number of rules per rule groups for all namespaces continues to be configured by `-ruler.max-rules-per-rule-group`, but now, this can be superseded by the new `-ruler.max-rules-per-rule-group-by-namespace` option on a per namespace basis. This new limit can be overridden using the overrides mechanism to be applied per-tenant. #8378
Expand All @@ -32,6 +33,7 @@
* [ENHANCEMENT] Alertmanager: Reloading config and templates no longer needs to hit the disk. #4967
* [ENHANCEMENT] Compactor: Added experimantal `-compactor.in-memory-tenant-meta-cache-size` option to set size of in-memory cache (in number of items) for parsed meta.json files. This can help when tenant has many meta.json files and their parsing before each compaction cycle is using a lot of CPU time. #8544
* [ENHANCEMENT] Distributor: Interrupt OTLP write request translation when context is canceled or has timed out. #8524
* [ENHANCEMENT] Ingester, store-gateway: optimised regular expression matching for patterns like `1.*|2.*|3.*|...|1000.*`. #8632
* [BUGFIX] Query-frontend: fix `-querier.max-query-lookback` enforcement when `-compactor.blocks-retention-period` is not set, and viceversa. #8388
* [BUGFIX] Ingester: fix sporadic `not found` error causing an internal server error if label names are queried with matchers during head compaction. #8391
* [BUGFIX] Ingester, store-gateway: fix case insensitive regular expressions not matching correctly some Unicode characters. #8391
Expand Down
4 changes: 2 additions & 2 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ require (
github.com/golang/snappy v0.0.4
github.com/google/gopacket v1.1.19
github.com/gorilla/mux v1.8.1
github.com/grafana/dskit v0.0.0-20240626184720-35810fdf1c6d
github.com/grafana/dskit v0.0.0-20240704113758-97b2aa96bcb1
github.com/grafana/e2e v0.1.2-0.20240118170847-db90b84177fc
github.com/hashicorp/golang-lru v1.0.2 // indirect
github.com/json-iterator/go v1.1.12
Expand Down Expand Up @@ -277,7 +277,7 @@ require (
)

// Using a fork of Prometheus with Mimir-specific changes.
replace github.com/prometheus/prometheus => github.com/grafana/mimir-prometheus v0.0.0-20240703103907-5131622acfee
replace github.com/prometheus/prometheus => github.com/grafana/mimir-prometheus v0.0.0-20240704133652-fb0cb30e280c

// Replace memberlist with our fork which includes some fixes that haven't been
// merged upstream yet:
Expand Down
8 changes: 4 additions & 4 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -515,8 +515,8 @@ github.com/grafana-tools/sdk v0.0.0-20220919052116-6562121319fc h1:PXZQA2WCxe85T
github.com/grafana-tools/sdk v0.0.0-20220919052116-6562121319fc/go.mod h1:AHHlOEv1+GGQ3ktHMlhuTUwo3zljV3QJbC0+8o2kn+4=
github.com/grafana/alerting v0.0.0-20240625192930-5b0553a572d3 h1:a7dtjsnqluU+1snjK+fCDbb/zj2b8STpNGsMm9bF92w=
github.com/grafana/alerting v0.0.0-20240625192930-5b0553a572d3/go.mod h1:DLj8frbtCaITljC2jc0L85JQViPF3mPfOSiYhm1osso=
github.com/grafana/dskit v0.0.0-20240626184720-35810fdf1c6d h1:CD8PWWX+9lYdgeMquSofmLErvCtk7jb+3/W/SH6oo/k=
github.com/grafana/dskit v0.0.0-20240626184720-35810fdf1c6d/go.mod h1:HvSf3uf8Ps2vPpzHeAFyZTdUcbVr+Rxpq1xcx7J/muc=
github.com/grafana/dskit v0.0.0-20240704113758-97b2aa96bcb1 h1:Hf7Swe4Vr+makVm+TSym1u0QMolxMQ1/L9gm1Pzdq5I=
github.com/grafana/dskit v0.0.0-20240704113758-97b2aa96bcb1/go.mod h1:UA1BG0yY/B7lTcdeqoud+3/TglKmPL88OM5qCeRs8BU=
github.com/grafana/e2e v0.1.2-0.20240118170847-db90b84177fc h1:BW+LjKJDz0So5LI8UZfW5neWeKpSkWqhmGjQFzcFfLM=
github.com/grafana/e2e v0.1.2-0.20240118170847-db90b84177fc/go.mod h1:JVmqPBe8A/pZWwRoJW5ZjyALeY5OXMzPl7LrVXOdZAI=
github.com/grafana/goautoneg v0.0.0-20240607115440-f335c04c58ce h1:WI1olbgS+sEl77qxEYbmt9TgRUz7iLqmjh8lYPpGlKQ=
Expand All @@ -525,8 +525,8 @@ github.com/grafana/gomemcache v0.0.0-20240229205252-cd6a66d6fb56 h1:X8IKQ0wu40wp
github.com/grafana/gomemcache v0.0.0-20240229205252-cd6a66d6fb56/go.mod h1:PGk3RjYHpxMM8HFPhKKo+vve3DdlPUELZLSDEFehPuU=
github.com/grafana/memberlist v0.3.1-0.20220714140823-09ffed8adbbe h1:yIXAAbLswn7VNWBIvM71O2QsgfgW9fRXZNR0DXe6pDU=
github.com/grafana/memberlist v0.3.1-0.20220714140823-09ffed8adbbe/go.mod h1:MS2lj3INKhZjWNqd3N0m3J+Jxf3DAOnAH9VT3Sh9MUE=
github.com/grafana/mimir-prometheus v0.0.0-20240703103907-5131622acfee h1:H6XAHAr8YzO61zZvmGmSWt8QFgkWHbXTWwxqCHf5Wy8=
github.com/grafana/mimir-prometheus v0.0.0-20240703103907-5131622acfee/go.mod h1:DjC1WWVnO+kFTzxrvJ6uS3SvbtoRiW4/lgLFUd49OPg=
github.com/grafana/mimir-prometheus v0.0.0-20240704133652-fb0cb30e280c h1:9oIh0utTogdAZlmlqIOuq5y9lFVtCPnk29AQa5ChufM=
github.com/grafana/mimir-prometheus v0.0.0-20240704133652-fb0cb30e280c/go.mod h1:DjC1WWVnO+kFTzxrvJ6uS3SvbtoRiW4/lgLFUd49OPg=
github.com/grafana/opentracing-contrib-go-stdlib v0.0.0-20230509071955-f410e79da956 h1:em1oddjXL8c1tL0iFdtVtPloq2hRPen2MJQKoAWpxu0=
github.com/grafana/opentracing-contrib-go-stdlib v0.0.0-20230509071955-f410e79da956/go.mod h1:qtI1ogk+2JhVPIXVc6q+NHziSmy2W5GbdQZFUHADCBU=
github.com/grafana/prometheus-alertmanager v0.25.1-0.20240625192351-66ec17e3aa45 h1:AJKOtDKAOg8XNFnIZSmqqqutoTSxVlRs6vekL2p2KEY=
Expand Down
2 changes: 2 additions & 0 deletions pkg/frontend/querymiddleware/astmapper/parallel.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,8 @@ var NonParallelFuncs = []string{
"absent",
"absent_over_time",
"histogram_quantile",
"limitk",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think limitk is safe to parallelize, just like topk/bottomk.

Maybe it's worth creating a new good-first-issue issue and let someone send a PR adding some tests for it?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Topk and bottok are not parallelizeable as well (see details in this internal ticket).

Why they're not mentioned here? Right now I don't remember all implementation details of how these functions are guaranteed not to be parallelized. I will investigate it further, but will not block this PR on this.

"limit_ratio",
"sort_desc",
"sort_by_label",
"sort_by_label_desc",
Expand Down
14 changes: 2 additions & 12 deletions pkg/storage/ingest/reader.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -304,12 +304,7 @@ func (r *PartitionReader) processNextFetchesUntilLagHonored(ctx context.Context,
}

if boff.Err() != nil {
// TODO should be moved to dskit's backoff
if ctx.Err() != nil {
return 0, context.Cause(ctx)
}

return 0, boff.Err()
return 0, boff.ErrCause()
}

// If it took less than the max desired lag to replay the partition
Expand All @@ -319,12 +314,7 @@ func (r *PartitionReader) processNextFetchesUntilLagHonored(ctx context.Context,
}
}

// TODO should be moved to dskit's backoff
if ctx.Err() != nil {
return 0, context.Cause(ctx)
}

return 0, boff.Err()
return 0, boff.ErrCause()
}

func filterOutErrFetches(fetches kgo.Fetches) kgo.Fetches {
Expand Down
11 changes: 10 additions & 1 deletion vendor/github.com/grafana/dskit/backoff/backoff.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

129 changes: 102 additions & 27 deletions vendor/github.com/prometheus/prometheus/model/labels/regexp.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading