move WAL tailing code from Prometheus to TSDB WAL package #606

cstyan · 2019-05-17T06:12:45Z

This is probably a bit of a pain to review, but moving the WAL watcher code from Prometheus to TSDB (into the WAL package) requires moving some code from the root tsdb package. Specifically the Watcher uses RefSeries and RefSample. Moving those requires a bunch of other code to be moved as well.

If anyone knows a nicer way to make the RefSeries/RefSamples change or has thoughts about the new package organization let me know, none of this is set in stone.

cc @tomwilkie @gouthamve

csmarchbanks

Generally I think moving all the record related code to a record package makes a lot of sense. I would try to limit code in that package to just types/methods directly interacting with a record though. Left a couple more specific comments around things you might be able to move out.

record/internal.go

record/tombstones.go

csmarchbanks · 2019-06-06T03:12:37Z

Thanks for all the responses, they helped me get my head around all this! As you said, the amount of code in the root package can be problematic to wrap your head around.

My main concern with the current state of this PR is that the record package appears to be random types with no obvious relation. I don't think it will help the code being more isolated/easier to wrap your head around.

After thinking about this for a couple hours:

I am pretty against memSeries being moved away from head.
- Perhaps have a slice of *memSeries next to the slice of RefSamples in headAppender instead of having RefSample to depend on memSeries?
Should the record code (and RefSample/RefSeries) live in wal? Are records used anywhere else? Otherwise I like them living in their own record package.
Perhaps tombstones should be in their own package? I don't feel too strongly about this either, just having Stone live wherever RefSample does would be fine by me too.

I am definitely curious what you and the maintainers think!

codesome · 2019-06-06T07:11:57Z

As I am not much familiar with WAL tailing yet, I am still wrapping my head about what the record package should be. But I agree with some points which @csmarchbanks said

memSeries looks closely tied with Head and would be good to keep it there.
If there are only a few things used from tombstones.go in the record, then duplicate only those constructs and not the entire file. Else tombstones as a separate package would make more sense than being in inside record. (I support the former)

I will take a better look before the weekend hits.

cstyan · 2019-06-07T22:16:47Z

I've moved the tombstones code to it's own package. Unfortunately keeping memSeries in head and have the records package RefSample not contain memSeries is going to be pretty messy.

cstyan · 2019-06-10T17:16:16Z

cc @csmarchbanks @codesome

csmarchbanks · 2019-06-10T18:41:05Z

I like the tombstone move. I think you could remove the memSeries from RefSample without it becoming too messy by keeping two slices in sync. What do you think of
memSeries.patch.txt? You would have to apply to master, not your branch.

krasi-georgiev · 2019-06-10T19:34:51Z

What is the problem you are trying to solve with this PR?

cstyan · 2019-06-10T20:29:42Z

@krasi-georgiev the code for tailing/watching the WAL currently lives in the remote write package in prometheus/prometheus. We think it makes more sense for it to live in the WAL package, which is in tsdb repo right now. Unfortunately the WAL tailing code relies on a couple of types that live in the root tsdb package, which would end up causing some circular dependency issues.

krasi-georgiev · 2019-06-12T12:39:10Z

hm , I see.
In that case need to spend some more time to look into this.

csmarchbanks

Did another pass tonight, generally I am liking the refactor and beginnings of pulling things out of the base package.

record/internal.go

wal/wal_watcher.go

wal/checkpoint.go

cstyan · 2019-07-05T15:25:51Z

@csmarchbanks thanks for the review and for the sync.Pool suggestion #606 (comment)

codesome

Not familiar with WAL Watcher, but other changes look fine.

head.go

record/internal.go

wal/watcher.go

csmarchbanks

Starting to look good from my perspective! Left another handful of comments, mostly pretty minor.

head.go

record/internal.go

wal/watcher.go

wal/watcher_test.go

cstyan · 2019-07-19T21:02:27Z

@csmarchbanks @codesome could use another review when you have some time

csmarchbanks

A couple more comments.

csmarchbanks · 2019-07-19T21:52:25Z

wal/watcher.go

+	SeriesReset(int)
+}
+
+type watcherMetrics struct {


I would prefer if this were WatcherMetrics to make it easier to pass around elsewhere.

csmarchbanks · 2019-07-19T21:53:26Z

wal/watcher.go

+}
+
+// NewWatcher creates a new WAL watcher for a given WriteTo.
+func NewWatcher(reg prometheus.Registerer, logger log.Logger, name string, writer WriteTo, walDir string) *Watcher {


This should take a WatcherMetrics, not a registerer, otherwise creating two watchers with the same registerer would panic.

Both NewWatcherMetrics and NewLiveReaderMetrics call reg.Register, not reg.MustRegister. But I guess the real question here is whether it's reasonable for consumers of the WAL Watcher to have to create and pass the metrics struct rather than just passing a Registerer.

True, reg.Register won't panic, but it also won't register the new metrics, so we would be blind for those queues/after a reload.

I think it is reasonable to have a consumer create and pass the metrics struct, just like NewLiveReader.

csmarchbanks · 2019-07-19T22:21:09Z

wal/watcher.go

+}
+
+// NewWatcher creates a new WAL watcher for a given WriteTo.
+func NewWatcher(reg prometheus.Registerer, logger log.Logger, name string, writer WriteTo, walDir string) *Watcher {


True, reg.Register won't panic, but it also won't register the new metrics, so we would be blind for those queues/after a reload.

I think it is reasonable to have a consumer create and pass the metrics struct, just like NewLiveReader.

csmarchbanks · 2019-07-19T22:24:31Z

wal/watcher.go

+	maxSegment int
+}
+
+func NewWatcherMetrics(reg prometheus.Registerer) *watcherMetrics {


Should these take a consumer and provide an Unregister so that if a consumer disappears it can be removed from the metrics?

We can't Unregister as all the Watcher metrics are under the same metric but just WithLabels. Both QueueManager and Watcher Stop delete their metrics via DeleteLabelValues.

codesome

LGTM. I hope I did not miss anything, PR is big.

tombstones/tombstones.go

wal/reader_test.go

codesome · 2019-07-26T11:22:18Z

Forgot to mention I haven't reviewed wal/watcher.go assuming that it was already reviewed when it was added to Prometheus.

Signed-off-by: Callum Styan <[email protected]>

something from Prometheus. Signed-off-by: Callum Styan <[email protected]>

Signed-off-by: Callum Styan <[email protected]>

manually creating a checkpoint dir and renaming files. Signed-off-by: Callum Styan <[email protected]>

Signed-off-by: Callum Styan <[email protected]>

after refactoring. Signed-off-by: Callum Styan <[email protected]>

Signed-off-by: Callum Styan <[email protected]>

refactor. Signed-off-by: Callum Styan <[email protected]>

Signed-off-by: Callum Styan <[email protected]>

the live reader, expose WriteTo interface. Signed-off-by: Callum Styan <[email protected]>

Signed-off-by: Callum Styan <[email protected]>

cstyan · 2019-07-29T19:05:06Z

thanks @codesome, the only new addition to watcher.go here is how it's metrics are initialized

edit: do we want to merge this before TSDB is moved in prometheus/prometheus? or after, and just apply the diff from here to the other repo?

than creating and registering in the constructor. Signed-off-by: Callum Styan <[email protected]>

csmarchbanks

LGTM!

bwplotka

It looks good overall - I agree with code movement, thanks for this!
However, I think there are some suggestions from my side:

PR is huge. Can we just move things around in one PR and add some extra logic (seriesSample) in another? It will help massively even if it's tedious ): e.g I don't understand really why this change.
tombstones package can be simplified due to nice clear name
some public methods/functions are missing comments.

bwplotka · 2019-08-13T08:03:18Z

compact.go

@@ -607,7 +608,7 @@ func (c *LeveledCompactor) write(dest string, meta *BlockMeta, blocks ...BlockRe
 	}

 	// Create an empty tombstones file.
-	if _, err := writeTombstoneFile(c.logger, tmp, newMemTombstones()); err != nil {
+	if _, err := tombstones.WriteTombstoneFile(c.logger, tmp, tombstones.NewMemTombstones()); err != nil {


Are we sure tombstones.WriteTombstoneFile needs to be WriteTombstoneFile etc? Why not just tombstones.WriteFile if we have clear package name?

same for NewMemTombstones

bwplotka · 2019-08-13T08:03:52Z

compact.go

 	err       error
 }

-func newCompactionSeriesSet(i IndexReader, c ChunkReader, t TombstoneReader, p index.Postings) *compactionSeriesSet {
+func newCompactionSeriesSet(i IndexReader, c ChunkReader, t tombstones.TombstoneReader, p index.Postings) *compactionSeriesSet {


ditto

Suggested change

func newCompactionSeriesSet(i IndexReader, c ChunkReader, t tombstones.TombstoneReader, p index.Postings) *compactionSeriesSet {

func newCompactionSeriesSet(i IndexReader, c ChunkReader, t tombstones.Reader, p index.Postings) *compactionSeriesSet {

bwplotka · 2019-08-13T08:04:26Z

head.go

 	// ErrOutOfBounds is returned if an appended sample is out of the
 	// writable time range.
 	ErrOutOfBounds = errors.New("out of bounds")

+	// ErrAmendSample is returned if an appended sample has the same timestamp


Why moving around?

bwplotka · 2019-08-13T08:05:52Z

head.go

 	//lint:ignore SA6002 safe to ignore and actually fixing it has some performance penalty.
 	h.appendPool.Put(b[:0])
 }

+func (h *Head) getSeriesBuffer() []*memSeries {


do we have to change this while just moving packages around?

bwplotka · 2019-08-13T08:08:15Z

tombstones/tombstones.go

@@ -30,7 +32,7 @@ import (
 	"github.com/prometheus/tsdb/fileutil"
 )

-const tombstoneFilename = "tombstones"
+const TombstoneFilename = "tombstones"


again, with tombstones package name you don't need tombstones prefix everywhere, we can simplify (:

bwplotka · 2019-08-13T08:08:53Z

tombstones/tombstones.go

@@ -54,8 +71,8 @@ type TombstoneReader interface {
 	Close() error
 }

-func writeTombstoneFile(logger log.Logger, dir string, tr TombstoneReader) (int64, error) {
-	path := filepath.Join(dir, tombstoneFilename)
+func WriteTombstoneFile(logger log.Logger, dir string, tr TombstoneReader) (int64, error) {


As it is exposed we need comment e.g for goDoc ): Same everywhere else for public functions we add.

codesome marked this pull request as ready for review May 31, 2019 12:38

cstyan force-pushed the callum-move-wal-watcher branch 3 times, most recently from 7dfb442 to 1caa80c Compare June 5, 2019 00:01

csmarchbanks reviewed Jun 5, 2019

View reviewed changes

record/internal.go Outdated Show resolved Hide resolved

record/internal.go Outdated Show resolved Hide resolved

record/internal.go Outdated Show resolved Hide resolved

record/tombstones.go Outdated Show resolved Hide resolved

krasi-georgiev mentioned this pull request Jun 6, 2019

Optionally compress the WAL using Snappy #609

Merged

cstyan force-pushed the callum-move-wal-watcher branch from 62729b5 to 0c895bd Compare June 7, 2019 22:08

cstyan force-pushed the callum-move-wal-watcher branch 2 times, most recently from ca248e3 to faf81f5 Compare June 27, 2019 15:26

cstyan force-pushed the callum-move-wal-watcher branch from faf81f5 to e944b75 Compare July 2, 2019 21:07

cstyan mentioned this pull request Jul 3, 2019

Compress WAL prometheus/prometheus#5592

Merged

cstyan force-pushed the callum-move-wal-watcher branch from e944b75 to d6ba13b Compare July 3, 2019 18:54

csmarchbanks reviewed Jul 4, 2019

View reviewed changes

record/internal.go Outdated Show resolved Hide resolved

record/internal.go Outdated Show resolved Hide resolved

wal/wal_watcher.go Outdated Show resolved Hide resolved

wal/checkpoint.go Outdated Show resolved Hide resolved

wal/checkpoint.go Outdated Show resolved Hide resolved

codesome reviewed Jul 8, 2019

View reviewed changes

head.go Outdated Show resolved Hide resolved

record/internal.go Outdated Show resolved Hide resolved

wal/watcher.go Show resolved Hide resolved

csmarchbanks reviewed Jul 8, 2019

View reviewed changes

cstyan force-pushed the callum-move-wal-watcher branch 3 times, most recently from f44209e to 476bbec Compare July 18, 2019 15:28

csmarchbanks reviewed Jul 19, 2019

View reviewed changes

codesome reviewed Jul 26, 2019

View reviewed changes

tombstones/tombstones.go Outdated Show resolved Hide resolved

wal/reader_test.go Show resolved Hide resolved

cstyan added 15 commits July 29, 2019 11:39

Move WAL Watcher from Prometheus to TSDB WAL package.

ce08521

Signed-off-by: Callum Styan <[email protected]>

Copy FromTime function from timestamp package so we don't have to vendor

643a5c9

something from Prometheus. Signed-off-by: Callum Styan <[email protected]>

WAL Watcher needs to take in and pass a Registerer to LiveReader.

b84a297

Signed-off-by: Callum Styan <[email protected]>

Call Checkpoint in TestReadCheckpointMultipleSegments instead of

6c4df84

manually creating a checkpoint dir and renaming files. Signed-off-by: Callum Styan <[email protected]>

Move tombstones to it's own package.

f453859

Signed-off-by: Callum Styan <[email protected]>

Fix some stuff I broke during rebase.

adfe7f4

Signed-off-by: Callum Styan <[email protected]>

Keep memSeries in head.go

f2c8171

Signed-off-by: Callum Styan <[email protected]>

We can just duplicate ErrNotFound to the record package.

1a4aba4

Signed-off-by: Callum Styan <[email protected]>

These functions aren't used anywhere within the record package anymore

fd51852

after refactoring. Signed-off-by: Callum Styan <[email protected]>

Change type names to remove record.Record... stutter.

323b5c4

Signed-off-by: Callum Styan <[email protected]>

Rename WALWatcher -> Watcher; fix creation/registration of it's metrics.

fe01392

Signed-off-by: Callum Styan <[email protected]>

Review fixes; mostly some things that don't need to be exported after

7e36b01

refactor. Signed-off-by: Callum Styan <[email protected]>

These comments don't need to change.

c392f57

Signed-off-by: Callum Styan <[email protected]>

Update WAL watcher from Prometheus repo, pass metrics structs around for

5264c56

the live reader, expose WriteTo interface. Signed-off-by: Callum Styan <[email protected]>

Don't export MemTombstones.

8c40bb9

Signed-off-by: Callum Styan <[email protected]>

cstyan force-pushed the callum-move-wal-watcher branch from 4b7a4fb to 8c40bb9 Compare July 29, 2019 18:56

cstyan requested a review from codesome August 2, 2019 23:12

Export WatcherMetrics struct and pass the struct to NewWatcher rather

3b20c64

than creating and registering in the constructor. Signed-off-by: Callum Styan <[email protected]>

csmarchbanks approved these changes Aug 8, 2019

View reviewed changes

codesome approved these changes Aug 13, 2019

View reviewed changes

bwplotka reviewed Aug 13, 2019

View reviewed changes

codesome mentioned this pull request Sep 10, 2019

Move WAL watcher code to tsdb/wal package. prometheus/prometheus#5999

Merged

	func newCompactionSeriesSet(i IndexReader, c ChunkReader, t tombstones.TombstoneReader, p index.Postings) *compactionSeriesSet {
	func newCompactionSeriesSet(i IndexReader, c ChunkReader, t tombstones.Reader, p index.Postings) *compactionSeriesSet {

move WAL tailing code from Prometheus to TSDB WAL package #606

Are you sure you want to change the base?

move WAL tailing code from Prometheus to TSDB WAL package #606

Conversation

cstyan commented May 17, 2019

csmarchbanks left a comment

Choose a reason for hiding this comment

csmarchbanks commented Jun 6, 2019 • edited Loading

codesome commented Jun 6, 2019

cstyan commented Jun 7, 2019

cstyan commented Jun 10, 2019

csmarchbanks commented Jun 10, 2019

krasi-georgiev commented Jun 10, 2019

cstyan commented Jun 10, 2019

krasi-georgiev commented Jun 12, 2019

csmarchbanks left a comment

Choose a reason for hiding this comment

cstyan commented Jul 5, 2019

codesome left a comment

Choose a reason for hiding this comment

csmarchbanks left a comment

Choose a reason for hiding this comment

cstyan commented Jul 19, 2019

csmarchbanks left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codesome left a comment

Choose a reason for hiding this comment

codesome commented Jul 26, 2019

cstyan commented Jul 29, 2019 • edited Loading

csmarchbanks left a comment

Choose a reason for hiding this comment

bwplotka left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

csmarchbanks commented Jun 6, 2019 •

edited

Loading

cstyan commented Jul 29, 2019 •

edited

Loading