From 12c7eab8bb94fd82b184c1c222200e37f2ca050a Mon Sep 17 00:00:00 2001 From: Ed Welch Date: Wed, 20 May 2020 11:38:32 -0400 Subject: [PATCH] Prep 1.5.0 release (#2098) * Updating the Changelog and Upgrade guide for 1.5.0 release. Signed-off-by: Ed Welch * Changing release number in all the docs Signed-off-by: Ed Welch --- CHANGELOG.md | 252 ++++++++++++++++++++++++++ README.md | 29 ++- docs/clients/promtail/installation.md | 2 +- docs/installation/docker.md | 18 +- docs/operations/loki-canary.md | 2 +- docs/operations/upgrade.md | 91 +++++++++- production/docker-compose.yaml | 4 +- 7 files changed, 369 insertions(+), 29 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 49f786e889916..85ac6c732839a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,255 @@ +## 1.5.0 (2020-05-20) + +It's been a busy month and a half since 1.4.0 was released, and a lot of new improvements have been added to Loki since! + +Be prepared for some configuration changes that may cause some bumps when upgrading, +we apologize for this but are always striving to reach the right compromise of code simplicity and user/operating experience. + +In this case we opted to keep a simplified configuration inline with Cortex rather than a more complicated and error prone internal config mapping or difficult to implement support for multiple config names for the same feature. + +This does result in breaking config changes for some configurations, however, these will fail fast and with the [list of diffs](https://cortexmetrics.io/docs/changelog/#config-file-breaking-changes) from the Cortex project should be quick to fix. + +### Important Notes + +**Be prepared for breaking config changes.** Loki 1.5.0 vendors cortex [v1.0.1-0.20200430170006-3462eb63f324](https://github.com/cortexproject/cortex/commit/3462eb63f324c649bbaa122933bc591b710f4e48), +there were substantial breaking config changes in Cortex 1.0 which standardized config options, and fixed typos. + +**The Loki docker image user has changed to no longer be root** + +Check the [upgrade guide](https://github.com/grafana/loki/blob/master/docs/operations/upgrade.md#150) for more detailed information on these changes. + +### Notable Features and Fixes + +There are quite a few we want to mention listed in order they were merged (mostly) + +* [1837](https://github.com/grafana/loki/pull/1837) **sandeepsukhani**: flush boltdb to object store + +This is perhaps the most exciting feature of 1.5.0, the first steps in removing a dependency on a separate index store! This feature is still very new and experimental, however, we want this to be the future for Loki. Only requiring just an object store. + +If you want to test this new feature, and help us find any bugs, check out the [docs](docs/operations/storage/boltdb-shipper.md) to learn more and get started. + +* [2073](https://github.com/grafana/loki/pull/2073) **slim-bean**: Loki: Allow configuring query_store_max_look_back_period when running a filesystem store and boltdb-shipper + +This is even more experimental than the previous feature mentioned however also pretty exciting for Loki users who use the filesystem storage. We can leverage changes made in [1837](https://github.com/grafana/loki/pull/1837) to now allow Loki to run in a clustered mode with individual filesystem stores! + +Please check out the last section in the [filesystem docs](docs/operations/storage/filesystem.md) for more details on how this works and how to use it! + +* [2095](https://github.com/grafana/loki/pull/2095) **cyriltovena**: Adds backtick for the quoted string token lexer. + +This will come as a big win to anyone who is writing complicated reqular expressions in either their Label matchers or Filter Expressions. Starting now you can use the backtick to encapsulate your regex **and not have to do any escaping of special characters!!** + +Examples: + +``` +{name="cassandra"} |~ `error=\w+` +{name!~`mysql-\d+`} +``` + +* [2055](https://github.com/grafana/loki/pull/2055) **aknuds1**: Chore: Fix spelling of per second in code + +This is technically a breaking change for anyone who wrote code to processes the new statistics output in the query result added in 1.4.0, we apologize to anyone in this situation but if we don't fix this kind of error now it will be there forever. +And at the same time we didn't feel it was appropriate to make any major api revision changes for such a new feature and simple change. We are always trying to use our best judgement in cases like this. + +* [2031](https://github.com/grafana/loki/pull/2031) **cyriltovena**: Improve protobuf serialization + +Thanks @cyriltovena for another big performance improvement in Loki, this time around protbuf's! + +* [2021](https://github.com/grafana/loki/pull/2021) **slim-bean**: Loki: refactor validation and improve error messages +* [2012](https://github.com/grafana/loki/pull/2012) **slim-bean**: Loki: Improve logging and add metrics to streams dropped by stream limit + +These two changes standardize the metrics used to report when a tenant hits a limit, now all discarded samples should be reported under `loki_discarded_samples_total` and you no longer need to also reference `cortex_discarded_samples_total`. +Additionally error messages were improved to help clients take better action when hitting limits. + +* [1970](https://github.com/grafana/loki/pull/1970) **cyriltovena**: Allow to aggregate binary operations. + +Another nice improvement to the query language which allows queries like this to work now: + +``` +sum by (job) (count_over_time({namespace="tns"}[5m] |= "level=error") / count_over_time({namespace="tns"}[5m])) +``` + +* [1713](https://github.com/grafana/loki/pull/1713) **adityacs**: Log error message for invalid checksum + +In the event something went wrong with a stored chunk, rather than fail the query we ignore the chunk and return the rest. + +* [2066](https://github.com/grafana/loki/pull/2066) **slim-bean**: Promtail: metrics stage can also count line bytes + +This is a nice extension to a previous feature which let you add a metric to count log lines per stream, you can now count log bytes per stream. + +Check out [this example](docs/clients/promtail/configuration.md#counter) to configure this in your promtail pipelines. + +* [1935](https://github.com/grafana/loki/pull/1935) **cyriltovena**: Support stdin target via flag instead of automatic detection. + +Third times a charm! With 1.4.0 we allowed sending logs directly to promtail via stdin, with 1.4.1 we released a patch for this feature which wasn't detecting stdin correctly on some operating systems. +Unfortunately after a few more bug reports it seems this change caused some more undesired side effects so we decided to not try to autodetect stdin at all, instead now you must pass the `--stdin` flag if you want Promtail to listen for logs on stdin. + +* [2076](https://github.com/grafana/loki/pull/2076) **cyriltovena**: Allows to pass inlined pipeline stages to the docker driver. +* [1906](https://github.com/grafana/loki/pull/1906) **cyriltovena**: Add no-file and keep-file log option for docker driver. + +The docker logging driver received a couple very nice updates, it's always been challenging to configure pipeline stages for the docker driver, with the first PR there are now a few easier ways to do this! +In the second PR we added config options to control keeping any log files on the host when using the docker logging driver, allowing you to run with no disk access if you would like, as well as allowing you to control keeping log files available after container restarts. + +** [1864](https://github.com/grafana/loki/pull/1864) **cyriltovena**: Sign helm package with GPG. + +We now GPG sign helm packages! + +### All Changes + +#### Loki + +* [2097](https://github.com/grafana/loki/pull/2097) **owen-d**: simplifies/updates some of our configuration examples +* [2095](https://github.com/grafana/loki/pull/2095) **cyriltovena**: Adds backtick for the quoted string token lexer. +* [2093](https://github.com/grafana/loki/pull/2093) **cyriltovena**: Fixes unit in stats request log. +* [2088](https://github.com/grafana/loki/pull/2088) **slim-bean**: Loki: allow no encoding/compression on chunks +* [2078](https://github.com/grafana/loki/pull/2078) **owen-d**: removes yolostring +* [2073](https://github.com/grafana/loki/pull/2073) **slim-bean**: Loki: Allow configuring query_store_max_look_back_period when running a filesystem store and boltdb-shipper +* [2064](https://github.com/grafana/loki/pull/2064) **cyriltovena**: Reverse entry iterator pool +* [2059](https://github.com/grafana/loki/pull/2059) **cyriltovena**: Recover from panic in http and grpc handlers. +* [2058](https://github.com/grafana/loki/pull/2058) **cyriltovena**: Fix a bug in range vector skipping data. +* [2055](https://github.com/grafana/loki/pull/2055) **aknuds1**: Chore: Fix spelling of per second in code +* [2046](https://github.com/grafana/loki/pull/2046) **gouthamve**: Fix bug in logql parsing that leads to crash. +* [2050](https://github.com/grafana/loki/pull/2050) **aknuds1**: Chore: Correct typo "per seconds" +* [2034](https://github.com/grafana/loki/pull/2034) **sandeepsukhani**: some metrics for measuring performance and failures in boltdb shipper +* [2031](https://github.com/grafana/loki/pull/2031) **cyriltovena**: Improve protobuf serialization +* [2030](https://github.com/grafana/loki/pull/2030) **adityacs**: Update loki to cortex master +* [2023](https://github.com/grafana/loki/pull/2023) **cyriltovena**: Support post requests in the frontend queryrange handler. +* [2021](https://github.com/grafana/loki/pull/2021) **slim-bean**: Loki: refactor validation and improve error messages +* [2019](https://github.com/grafana/loki/pull/2019) **slim-bean**: make `loki_ingester_memory_streams` Gauge per tenant. +* [2012](https://github.com/grafana/loki/pull/2012) **slim-bean**: Loki: Improve logging and add metrics to streams dropped by stream limit +* [2010](https://github.com/grafana/loki/pull/2010) **cyriltovena**: Update lz4 library to latest to ensure deterministic output. +* [2001](https://github.com/grafana/loki/pull/2001) **sandeepsukhani**: table client for boltdb shipper to enforce retention +* [1995](https://github.com/grafana/loki/pull/1995) **sandeepsukhani**: make boltdb shipper singleton and some other minor refactoring +* [1987](https://github.com/grafana/loki/pull/1987) **slim-bean**: Loki: Add a missing method to facade which is called by the metrics storage client in cortex +* [1982](https://github.com/grafana/loki/pull/1982) **cyriltovena**: Update cortex to latest. +* [1977](https://github.com/grafana/loki/pull/1977) **cyriltovena**: Ensure trace propagation in our logs. +* [1976](https://github.com/grafana/loki/pull/1976) **slim-bean**: incorporate some better defaults into table-manager configs +* [1975](https://github.com/grafana/loki/pull/1975) **slim-bean**: Update cortex vendoring to latest master +* [1970](https://github.com/grafana/loki/pull/1970) **cyriltovena**: Allow to aggregate binary operations. +* [1965](https://github.com/grafana/loki/pull/1965) **slim-bean**: Loki: Adds an `interval` paramater to query_range queries allowing a sampling of events to be returned based on the provided interval +* [1964](https://github.com/grafana/loki/pull/1964) **owen-d**: chunk bounds metric now records 8h range in 1h increments +* [1963](https://github.com/grafana/loki/pull/1963) **cyriltovena**: Improve the local config to work locally and inside docker. +* [1961](https://github.com/grafana/loki/pull/1961) **jpmcb**: [Bug] Workaround for broken etcd gomod import +* [1958](https://github.com/grafana/loki/pull/1958) **owen-d**: chunk lifespan histogram +* [1956](https://github.com/grafana/loki/pull/1956) **sandeepsukhani**: update cortex to latest master +* [1953](https://github.com/grafana/loki/pull/1953) **jpmcb**: Go mod: explicit golang.org/x/net replace +* [1950](https://github.com/grafana/loki/pull/1950) **cyriltovena**: Fixes case handling in regex simplification. +* [1949](https://github.com/grafana/loki/pull/1949) **SerialVelocity**: [Loki]: Cleanup dockerfile +* [1946](https://github.com/grafana/loki/pull/1946) **slim-bean**: Loki Update the cut block size counter when creating a memchunk from byte slice +* [1939](https://github.com/grafana/loki/pull/1939) **owen-d**: adds config validation, similar to cortex +* [1916](https://github.com/grafana/loki/pull/1916) **cyriltovena**: Add cap_net_bind_service linux capabilities to Loki. +* [1914](https://github.com/grafana/loki/pull/1914) **owen-d**: only fetches one chunk per series in /series +* [1875](https://github.com/grafana/loki/pull/1875) **owen-d**: support `match[]` encoding +* [1869](https://github.com/grafana/loki/pull/1869) **pstibrany**: Update Cortex to latest master +* [1846](https://github.com/grafana/loki/pull/1846) **owen-d**: Sharding optimizations I: AST mapping +* [1838](https://github.com/grafana/loki/pull/1838) **cyriltovena**: Move default port for Loki to 3100 everywhere. +* [1837](https://github.com/grafana/loki/pull/1837) **sandeepsukhani**: flush boltdb to object store +* [1834](https://github.com/grafana/loki/pull/1834) **Mario-Hofstaetter**: Loki/Change local storage directory to /loki/ and fix permissions (#1833) +* [1819](https://github.com/grafana/loki/pull/1819) **cyriltovena**: Adds a counter for total flushed chunks per reason. +* [1816](https://github.com/grafana/loki/pull/1816) **sdojjy**: loki can not be started with loki-local-config.yaml +* [1810](https://github.com/grafana/loki/pull/1810) **cyriltovena**: Optimize empty filter queries. +* [1809](https://github.com/grafana/loki/pull/1809) **cyriltovena**: Test stats memchunk +* [1804](https://github.com/grafana/loki/pull/1804) **pstibrany**: Convert Loki modules to services +* [1799](https://github.com/grafana/loki/pull/1799) **pstibrany**: loki: update Cortex to master +* [1798](https://github.com/grafana/loki/pull/1798) **adityacs**: Support configurable maximum of the limits parameter +* [1713](https://github.com/grafana/loki/pull/1713) **adityacs**: Log error message for invalid checksum +* [1706](https://github.com/grafana/loki/pull/1706) **cyriltovena**: Non-root user docker image for Loki. + +#### Logcli +* [2027](https://github.com/grafana/loki/pull/2027) **pstibrany**: logcli: Query needs to be stored into url.RawQuery, and not url.Path +* [2000](https://github.com/grafana/loki/pull/2000) **cyriltovena**: Improve URL building in the logcli to strip trailing /. +* [1922](https://github.com/grafana/loki/pull/1922) **bavarianbidi**: logcli: org-id/tls-skip-verify set via env var +* [1861](https://github.com/grafana/loki/pull/1861) **yeya24**: Support series API in logcli +* [1850](https://github.com/grafana/loki/pull/1850) **chrischdi**: BugFix: Fix logcli client to use OrgID in LiveTail +* [1814](https://github.com/grafana/loki/pull/1814) **cyriltovena**: Logcli remote storage. +* [1712](https://github.com/grafana/loki/pull/1712) **rfratto**: clarify logcli commands and output + +#### Promtail +* [2069](https://github.com/grafana/loki/pull/2069) **slim-bean**: Promtail: log at debug level when nothing matches the specified path for a file target +* [2066](https://github.com/grafana/loki/pull/2066) **slim-bean**: Promtail: metrics stage can also count line bytes +* [2049](https://github.com/grafana/loki/pull/2049) **adityacs**: Fix promtail client default values +* [2075](https://github.com/grafana/loki/pull/2075) **cyriltovena**: Fixes a panic in dry-run when using external labels. +* [2026](https://github.com/grafana/loki/pull/2026) **adityacs**: Targets not required in promtail config +* [2004](https://github.com/grafana/loki/pull/2004) **cyriltovena**: Adds config to disable HTTP and GRPC server in Promtail. +* [1935](https://github.com/grafana/loki/pull/1935) **cyriltovena**: Support stdin target via flag instead of automatic detection. +* [1920](https://github.com/grafana/loki/pull/1920) **alexanderGalushka**: feat: tms readiness check bypass implementation +* [1894](https://github.com/grafana/loki/pull/1894) **cyriltovena**: Fixes possible panic in json pipeline stage. +* [1865](https://github.com/grafana/loki/pull/1865) **adityacs**: Fix flaky promtail test +* [1815](https://github.com/grafana/loki/pull/1815) **adityacs**: Log error message when source does not exist in extracted values +* [1627](https://github.com/grafana/loki/pull/1627) **rfratto**: Proposal: Promtail Push API + +#### Docker Driver +* [2076](https://github.com/grafana/loki/pull/2076) **cyriltovena**: Allows to pass inlined pipeline stages to the docker driver. +* [2054](https://github.com/grafana/loki/pull/2054) **bkmit**: Docker driver: Allow to provision external pipeline files to plugin +* [1906](https://github.com/grafana/loki/pull/1906) **cyriltovena**: Add no-file and keep-file log option for docker driver. +* [1903](https://github.com/grafana/loki/pull/1903) **cyriltovena**: Log docker driver config map. + +#### FluentD +* [2074](https://github.com/grafana/loki/pull/2074) **osela**: fluentd plugin: support placeholders in tenant field +* [2006](https://github.com/grafana/loki/pull/2006) **Skeen**: fluent-plugin-loki: Restructuring and CI +* [1909](https://github.com/grafana/loki/pull/1909) **jgehrcke**: fluentd loki plugin README: add note about labels +* [1853](https://github.com/grafana/loki/pull/1853) **wardbekker**: bump gem version +* [1811](https://github.com/grafana/loki/pull/1811) **JamesJJ**: Error handling: Show data stream at "debug" level, not "warn" + +#### Fluent Bit +* [2040](https://github.com/grafana/loki/pull/2040) **avii-ridge**: Add extraOutputs variable to support multiple outputs for fluent-bit +* [1915](https://github.com/grafana/loki/pull/1915) **DirtyCajunRice**: Fix fluent-bit metrics +* [1890](https://github.com/grafana/loki/pull/1890) **dottedmag**: fluentbit: JSON encoding: avoid base64 encoding of []byte inside other slices +* [1791](https://github.com/grafana/loki/pull/1791) **cyriltovena**: Improve fluentbit logfmt. + +#### Ksonnet +* [1980](https://github.com/grafana/loki/pull/1980) **cyriltovena**: Log slow query from the frontend by default in ksonnet. + +##### Mixins +* [2080](https://github.com/grafana/loki/pull/2080) **beorn7**: mixin: Accept suffixes to pod name in instance labels +* [2044](https://github.com/grafana/loki/pull/2044) **slim-bean**: Dashboards: fixes the cpu usage graphs +* [2043](https://github.com/grafana/loki/pull/2043) **joe-elliott**: Swapped to container restarts over terminated reasons +* [2041](https://github.com/grafana/loki/pull/2041) **slim-bean**: Dashboard: Loki Operational improvements +* [1934](https://github.com/grafana/loki/pull/1934) **tomwilkie**: Put loki-mixin and promtail-mixin dashboards in a folder. +* [1913](https://github.com/grafana/loki/pull/1913) **tomwilkie**: s/dashboards/grafanaDashboards. + +#### Helm +* [2038](https://github.com/grafana/loki/pull/2038) **oke-py**: Docs: update Loki Helm Chart document to support Helm 3 +* [2015](https://github.com/grafana/loki/pull/2015) **etashsingh**: Change image tag from 1.4.1 to 1.4.0 in Helm chart +* [1981](https://github.com/grafana/loki/pull/1981) **sshah90**: added extraCommandlineArgs in values file +* [1967](https://github.com/grafana/loki/pull/1967) **rdxmb**: helm chart: add missing line feed +* [1898](https://github.com/grafana/loki/pull/1898) **stefanandres**: [helm loki/promtail] make UpdateStrategy configurable +* [1871](https://github.com/grafana/loki/pull/1871) **stefanandres**: [helm loki/promtail] Add systemd-journald example with extraMount, extraVolumeMount +* [1864](https://github.com/grafana/loki/pull/1864) **cyriltovena**: Sign helm package with GPG. +* [1825](https://github.com/grafana/loki/pull/1825) **polar3130**: Helm/loki-stack: refresh default grafana.image.tag to 6.7.0 +* [1817](https://github.com/grafana/loki/pull/1817) **bclermont**: Helm chart: Prevent prometheus to scrape both services + +#### Loki Canary +* [1891](https://github.com/grafana/loki/pull/1891) **joe-elliott**: Addition of a `/suspend` endpoint to Loki Canary + +#### Docs +* [2056](https://github.com/grafana/loki/pull/2056) **cyriltovena**: Update api.md +* [2014](https://github.com/grafana/loki/pull/2014) **jsoref**: Spelling +* [1999](https://github.com/grafana/loki/pull/1999) **oddlittlebird**: Docs: Added labels content +* [1974](https://github.com/grafana/loki/pull/1974) **rfratto**: fix stores for chunk and index in documentation for period_config +* [1966](https://github.com/grafana/loki/pull/1966) **oddlittlebird**: Docs: Update docker.md +* [1951](https://github.com/grafana/loki/pull/1951) **cstyan**: Move build from source instructions to root readme. +* [1945](https://github.com/grafana/loki/pull/1945) **FlorianLudwig**: docs: version pin the docker image in docker-compose +* [1925](https://github.com/grafana/loki/pull/1925) **wardbekker**: Clarified that the api push path needs to be specified. +* [1905](https://github.com/grafana/loki/pull/1905) **sshah90**: updating typo for end time parameter in api docs +* [1888](https://github.com/grafana/loki/pull/1888) **slim-bean**: docs: cleaning up the comments for the cache_config, default_validity option +* [1887](https://github.com/grafana/loki/pull/1887) **slim-bean**: docs: Adding a config change in release 1.4 upgrade doc, updating readme with new doc links +* [1881](https://github.com/grafana/loki/pull/1881) **cyriltovena**: Add precision about the range notation for LogQL. +* [1879](https://github.com/grafana/loki/pull/1879) **slim-bean**: docs: update promtail docs for backoff +* [1873](https://github.com/grafana/loki/pull/1873) **owen-d**: documents frontend worker +* [1870](https://github.com/grafana/loki/pull/1870) **ushuz**: Docs: Keep plugin install command example in one line +* [1856](https://github.com/grafana/loki/pull/1856) **slim-bean**: docs: tweak the doc section of the readme a little +* [1852](https://github.com/grafana/loki/pull/1852) **slim-bean**: docs: clean up schema recommendations +* [1843](https://github.com/grafana/loki/pull/1843) **vishesh92**: Docs: Update configuration docs for redis + +#### Build +* [2042](https://github.com/grafana/loki/pull/2042) **rfratto**: Fix drone +* [2009](https://github.com/grafana/loki/pull/2009) **cyriltovena**: Adds :delegated flags to speed up build experience on MacOS. +* [1942](https://github.com/grafana/loki/pull/1942) **owen-d**: delete tag script filters by prefix instead of substring +* [1918](https://github.com/grafana/loki/pull/1918) **slim-bean**: build: This Dockerfile is a remnant from a long time ago, not needed. +* [1911](https://github.com/grafana/loki/pull/1911) **slim-bean**: build: push images for `k` branches +* [1849](https://github.com/grafana/loki/pull/1849) **cyriltovena**: Pin helm version in circle-ci helm testing workflow. + + ## 1.4.1 (2020-04-06) We realized after the release last week that piping data into promtail was not working on Linux or Windows, this should fix this issue for both platforms: diff --git a/README.md b/README.md index 2779ada7f55a0..c713ee8c2f3ec 100644 --- a/README.md +++ b/README.md @@ -29,11 +29,9 @@ Loki differs from Prometheus by focusing on logs instead of metrics, and deliver ## Getting started -* [Installing Loki](https://github.com/grafana/loki/tree/v1.4.1/docs/installation/README.md) -* [Installing -Promtail](https://github.com/grafana/loki/tree/v1.4.1/docs/clients/promtail/installation.md) -* [Getting -Started Guide](https://github.com/grafana/loki/tree/v1.4.1/docs/getting-started/README.md) +* [Installing Loki](https://github.com/grafana/loki/tree/v1.5.0/docs/installation/README.md) +* [Installing Promtail](https://github.com/grafana/loki/tree/v1.5.0/docs/clients/promtail/installation.md) +* [Getting Started Guide](https://github.com/grafana/loki/tree/v1.5.0/docs/getting-started/README.md) ## Upgrading @@ -42,6 +40,7 @@ Started Guide](https://github.com/grafana/loki/tree/v1.4.1/docs/getting-started/ ### Documentation * [master](./docs/README.md) +* [v1.5.0](https://github.com/grafana/loki/tree/v1.5.0/docs/README.md) * [v1.4.1](https://github.com/grafana/loki/tree/v1.4.1/docs/README.md) * [v1.4.0](https://github.com/grafana/loki/tree/v1.4.0/docs/README.md) * [v1.3.0](https://github.com/grafana/loki/tree/v1.3.0/docs/README.md) @@ -49,18 +48,18 @@ Started Guide](https://github.com/grafana/loki/tree/v1.4.1/docs/getting-started/ * [v1.1.0](https://github.com/grafana/loki/tree/v1.1.0/docs/README.md) * [v1.0.0](https://github.com/grafana/loki/tree/v1.0.0/docs/README.md) -Commonly used sections (from the latest release v1.4.1): +Commonly used sections (from the latest release v1.5.0): -- [API documentation](https://github.com/grafana/loki/tree/v1.4.1/docs/api.md) for alternative ways of getting logs into Loki. +- [API documentation](https://github.com/grafana/loki/tree/v1.5.0/docs/api.md) for alternative ways of getting logs into Loki. - [Labels](https://github.com/grafana/loki/blob/master/docs/getting-started/labels.md) -- [Operations](https://github.com/grafana/loki/tree/v1.4.1/docs/operations) for important aspects of running Loki. -- [Promtail](https://github.com/grafana/loki/tree/v1.4.1/docs/clients/promtail) is an agent which can tail your log files and push them to Loki. -- [Pipelines](https://github.com/grafana/loki/tree/v1.4.1/docs/clients/promtail/pipelines.md) for detailed log processing pipeline documentation -- [Docker Logging Driver](https://github.com/grafana/loki/tree/v1.4.1/docs/clients/docker-driver) is a docker plugin to send logs directly to Loki from Docker containers. -- [LogCLI](https://github.com/grafana/loki/tree/v1.4.1/docs/getting-started/logcli.md) on how to query your logs without Grafana. -- [Loki Canary](https://github.com/grafana/loki/tree/v1.4.1/docs/operations/loki-canary.md) for monitoring your Loki installation for missing logs. -- [Troubleshooting](https://github.com/grafana/loki/tree/v1.4.1/docs/getting-started/troubleshooting.md) for help around frequent error messages. -- [Loki in Grafana](https://github.com/grafana/loki/tree/v1.4.1/docs/getting-started/grafana.md) for how to set up a Loki datasource in Grafana and query your logs. +- [Operations](https://github.com/grafana/loki/tree/v1.5.0/docs/operations) for important aspects of running Loki. +- [Promtail](https://github.com/grafana/loki/tree/v1.5.0/docs/clients/promtail) is an agent which can tail your log files and push them to Loki. +- [Pipelines](https://github.com/grafana/loki/tree/v1.5.0/docs/clients/promtail/pipelines.md) for detailed log processing pipeline documentation +- [Docker Logging Driver](https://github.com/grafana/loki/tree/v1.5.0/docs/clients/docker-driver) is a docker plugin to send logs directly to Loki from Docker containers. +- [LogCLI](https://github.com/grafana/loki/tree/v1.5.0/docs/getting-started/logcli.md) on how to query your logs without Grafana. +- [Loki Canary](https://github.com/grafana/loki/tree/v1.5.0/docs/operations/loki-canary.md) for monitoring your Loki installation for missing logs. +- [Troubleshooting](https://github.com/grafana/loki/tree/v1.5.0/docs/getting-started/troubleshooting.md) for help around frequent error messages. +- [Loki in Grafana](https://github.com/grafana/loki/tree/v1.5.0/docs/getting-started/grafana.md) for how to set up a Loki datasource in Grafana and query your logs. ## Getting Help diff --git a/docs/clients/promtail/installation.md b/docs/clients/promtail/installation.md index 7dae8efd31659..191e3efa870b7 100644 --- a/docs/clients/promtail/installation.md +++ b/docs/clients/promtail/installation.md @@ -12,7 +12,7 @@ Every release includes binaries for Promtail which can be found on the ```bash # modify tag to most recent version -$ docker pull grafana/promtail:1.4.1 +$ docker pull grafana/promtail:1.5.0 ``` ## Helm diff --git a/docs/installation/docker.md b/docs/installation/docker.md index 82be1cec37273..850ae4b3eedd5 100644 --- a/docs/installation/docker.md +++ b/docs/installation/docker.md @@ -15,10 +15,10 @@ For production, we recommend Tanka or Helm. Copy and paste the commands below into your command line. ```bash -wget https://raw.githubusercontent.com/grafana/loki/v1.4.1/cmd/loki/loki-local-config.yaml -O loki-config.yaml -docker run -v $(pwd):/mnt/config -p 3100:3100 grafana/loki:1.4.1 -config.file=/mnt/config/loki-config.yaml -wget https://raw.githubusercontent.com/grafana/loki/v1.4.1/cmd/promtail/promtail-docker-config.yaml -O promtail-config.yaml -docker run -v $(pwd):/mnt/config -v /var/log:/var/log grafana/promtail:1.4.1 -config.file=/mnt/config/promtail-config.yaml +wget https://raw.githubusercontent.com/grafana/loki/v1.5.0/cmd/loki/loki-local-config.yaml -O loki-config.yaml +docker run -v $(pwd):/mnt/config -p 3100:3100 grafana/loki:1.5.0 -config.file=/mnt/config/loki-config.yaml +wget https://raw.githubusercontent.com/grafana/loki/v1.5.0/cmd/promtail/promtail-docker-config.yaml -O promtail-config.yaml +docker run -v $(pwd):/mnt/config -v /var/log:/var/log grafana/promtail:1.5.0 -config.file=/mnt/config/promtail-config.yaml ``` When finished, loki-config.yaml and promtail-config.yaml are downloaded in the directory you chose. Docker containers are running Loki and Promtail using those config files. @@ -31,10 +31,10 @@ Copy and paste the commands below into your terminal. Note that you will need to ```bash cd "" -wget https://raw.githubusercontent.com/grafana/loki/v1.4.1/cmd/loki/loki-local-config.yaml -O loki-config.yaml -docker run -v :/mnt/config -p 3100:3100 grafana/loki:1.4.1 --config.file=/mnt/config/loki-config.yaml -wget https://raw.githubusercontent.com/grafana/loki/v1.4.1/cmd/promtail/promtail-docker-config.yaml -O promtail-config.yaml -docker run -v :/mnt/config -v /var/log:/var/log grafana/promtail:1.4.1 --config.file=/mnt/config/promtail-config.yaml +wget https://raw.githubusercontent.com/grafana/loki/v1.5.0/cmd/loki/loki-local-config.yaml -O loki-config.yaml +docker run -v :/mnt/config -p 3100:3100 grafana/loki:1.5.0 --config.file=/mnt/config/loki-config.yaml +wget https://raw.githubusercontent.com/grafana/loki/v1.5.0/cmd/promtail/promtail-docker-config.yaml -O promtail-config.yaml +docker run -v :/mnt/config -v /var/log:/var/log grafana/promtail:1.5.0 --config.file=/mnt/config/promtail-config.yaml ``` When finished, loki-config.yaml and promtail-config.yaml are downloaded in the directory you chose. Docker containers are running Loki and Promtail using those config files. @@ -44,6 +44,6 @@ Navigate to http://localhost:3100/metrics to view the output. ## Install with Docker Compose ```bash -$ wget https://raw.githubusercontent.com/grafana/loki/v1.4.1/production/docker-compose.yaml -O docker-compose.yaml +$ wget https://raw.githubusercontent.com/grafana/loki/v1.5.0/production/docker-compose.yaml -O docker-compose.yaml $ docker-compose -f docker-compose.yaml up ``` diff --git a/docs/operations/loki-canary.md b/docs/operations/loki-canary.md index 7aa31373b6ea1..5709b2f756d77 100644 --- a/docs/operations/loki-canary.md +++ b/docs/operations/loki-canary.md @@ -67,7 +67,7 @@ Loki Canary is also provided as a Docker container image: ```bash # change tag to the most recent release -$ docker pull grafana/loki-canary:1.4.1 +$ docker pull grafana/loki-canary:1.5.0 ``` ### Kubernetes diff --git a/docs/operations/upgrade.md b/docs/operations/upgrade.md index 9cf4deb10a218..bbd142fd41e05 100644 --- a/docs/operations/upgrade.md +++ b/docs/operations/upgrade.md @@ -8,6 +8,10 @@ On this page we will document any upgrade issues/gotchas/considerations we are a ## 1.5.0 +Note: The required upgrade path outlined for version 1.4.0 below is still true for moving to 1.5.0 from any release older than 1.4.0 (e.g. 1.3.0->1.5.0 needs to also look at the 1.4.0 upgrade requirements). + +### Breaking config changes! + Loki 1.5.0 vendors Cortex v1.0.0 (congratulations!), which has a [massive list of changes](https://cortexmetrics.io/docs/changelog/#1-0-0-2020-04-02). While changes in the command line flags affect Loki as well, we usually recommend people to use configuration file instead. @@ -16,6 +20,92 @@ Cortex has done lot of cleanup in the configuration files, and you are strongly Following fields were removed from YAML configuration completely: `claim_on_rollout` (always true), `normalise_tokens` (always true). +#### Test Your Config + +To see if your config needs to change, one way to quickly test is to download a 1.5.0 (or newer) binary from the [release page](https://github.com/grafana/loki/releases/tag/v1.5.0) + +Then run the binary providing your config file `./loki-linux-amd64 -config.file=myconfig.yaml` + +If there are configs which are no longer valid you will see errors immediately: + +```shell +./loki-linux-amd64 -config.file=loki-local-config.yaml +failed parsing config: loki-local-config.yaml: yaml: unmarshal errors: + line 35: field dynamodbconfig not found in type aws.StorageConfig +``` + +Referencing the [list of diffs](https://cortexmetrics.io/docs/changelog/#config-file-breaking-changes) I can see this config changed: + +```diff +- dynamodbconfig: ++ dynamodb: +``` + +Also several other AWS related configs changed and would need to udpate those as well. + + +### Loki Docker Image User and File Location Changes + +To improve security concerns, in 1.5.0 the Docker container no longer runs the loki process as `root` and instead the process runs as user `loki` with UID `10001` and GID `10001` + +This may affect people in a couple ways: + +#### Loki Port + +If you are running Loki with a config that opens a port number above 1000 (which is the default, 3100 for HTTP and 9095 for GRPC) everything should work fine in regards to ports. + +If you are running Loki with a config that opens a port number less than 1000 Linux normally requires root permissions to do this, HOWEVER in the Docker container we run `setcap cap_net_bind_service=+ep /usr/bin/loki` + +This capability lets the loki process bind to a port less than 1000 when run as a non root user. + +Not every environment will allow this capability however, it's possible to restrict this capability in linux. If this restriction is in place, you will be forced to run Loki with a config that has HTTP and GRPC ports above 1000. + +#### Filesystem + +**Please note the location Loki is looking for files with the provided config in the docker image has changed** + +In 1.4.0 and earlier the included config file in the docker container was using directories: + +``` +/tmp/loki/index +/tmp/loki/chunks +``` + +In 1.5.0 this has changed: + +``` +/loki/index +/loki/chunks +``` + +This will mostly affect anyone using docker-compose or docker to run Loki and are specifying a volume to persist storage. + +**There are two concerns to track here, one is the correct ownership of the files and the other is making sure your mounts updated to the new location.** + +One possible upgrade path would look like this: + +If I were running Loki with this command `docker run -d --name=loki --mount source=loki-data,target=/tmp/loki -p 3100:3100 grafana/loki:1.4.0` + +This would mount a docker volume named `loki-data` to the `/temp/loki` folder which is where Loki will persist the `index` and `chunks` folder in 1.4.0 + +To move to 1.5.0 I can do the following (please note that your container names and paths and volumes etc may be different): + +``` +docker stop loki +docker rm loki +docker run --rm --name="loki-perm" -it --mount source=loki-data,target=/mnt ubuntu /bin/bash +cd /mnt +chown -R 10001:10001 ./* +exit +docker run -d --name=loki --mount source=loki-data,target=/loki -p 3100:3100 grafana/loki:1.5.0 +``` + +Notice the change in the `target=/loki` for 1.5.0 to the new data directory location specified in the [included Loki config file](../../cmd/loki/loki-docker-config.yaml). + +The intermediate step of using an ubuntu image to change the ownership of the Loki files to the new user might not be necessary if you can easily access these files to run the `chown` command directly. +That is if you have access to `/var/lib/docker/volumes` or if you mounted to a different local filesystem directory, you can change the ownership directly without using a container. + + ## 1.4.0 Loki 1.4.0 vendors Cortex v0.7.0-rc.0 which contains [several breaking config changes](https://github.com/cortexproject/cortex/blob/v0.7.0-rc.0/CHANGELOG.md). @@ -84,4 +174,3 @@ If you attempt to add a v1.4.0 ingester to a ring created by Loki v1.2.0 or olde This will result in distributors failing to write and a general ingestion failure for the system. If this happens to you, you will want to rollback your deployment immediately. You need to remove the v1.4.0 ingester from the ring ASAP, this should allow the existing ingesters to re-insert their tokens. You will also want to remove any v1.4.0 distributors as they will not understand the old ring either and will fail to send traffic. - diff --git a/production/docker-compose.yaml b/production/docker-compose.yaml index 80edcd8651e52..b31014645673e 100644 --- a/production/docker-compose.yaml +++ b/production/docker-compose.yaml @@ -5,7 +5,7 @@ networks: services: loki: - image: grafana/loki:1.4.1 + image: grafana/loki:1.5.0 ports: - "3100:3100" command: -config.file=/etc/loki/local-config.yaml @@ -13,7 +13,7 @@ services: - loki promtail: - image: grafana/promtail:1.4.1 + image: grafana/promtail:1.5.0 volumes: - /var/log:/var/log command: -config.file=/etc/promtail/docker-config.yaml