First dumb of logic to get route index running on the journal #3743
Conversation
| @@ -1190,7 +1190,7 @@ public Runnable save(AccordCommandStore commandStore, TxnId txnId, @Nullable Com | |||
| return null; | |||
| } | |||
|
|
|||
| return commandStore.appendToKeyspace(txnId, value); | |||
| return null; | |||
There was a problem hiding this comment.
to lower the diff I didn't want to rewrite the whole cache logic... null is the only thing possible to return now
| @@ -126,6 +126,7 @@ public Collection<StaticSegment<JournalKey, V>> compact(Collection<StaticSegment | |||
| while ((advanced = reader.advance()) && reader.key().equals(key)); | |||
|
|
|||
| if (advanced) readers.offer(reader); // there is more to this reader, but not with this key | |||
| else reader.close(); | |||
There was a problem hiding this comment.
this leak detection was caught by org.apache.cassandra.index.accord.RouteIndexTest#test. The fs would run out of space and when you looked it had very few files and only ~40mb of data, yet 4g was allocated! The root cause was that each time we open a reader we create a new channel and didn't close it, so we could never purge segments.
| @@ -248,6 +248,8 @@ public static IVerbHandler<? extends Reply> responseHandlerOrNoop() | |||
|
|
|||
| public synchronized static void startup(NodeId tcmId) | |||
| { | |||
| if (instance != null) | |||
There was a problem hiding this comment.
safety check, I hit an issue and saw we didn't detect this so just added to be safer.
| durableBefores.put(safeStore.commandStore().id(), safeStore.durableBefore()); | ||
| } | ||
| })); | ||
| if (node.commandStores().all().length > 0) |
There was a problem hiding this comment.
in my tests we are testing journal but we don't have any tables marked for accord, so we had 0 stores... compaction tests were also impacted as we could have data but due to us dropping accord we no longer have stores
| import org.apache.cassandra.utils.ByteArrayUtil; | ||
| import org.apache.cassandra.utils.ObjectSizes; | ||
|
|
||
| public class IndexRange implements Comparable<IndexRange>, IMeasurableMemory |
There was a problem hiding this comment.
this was just extracted from the previous impl and put to the top level. With journal and tables both touching this, it was easier to make top level rather than an inner class
|
|
||
| package org.apache.cassandra.concurrent; | ||
|
|
||
| public class ForwardingExecutorFactory implements ExecutorFactory |
There was a problem hiding this comment.
used by RouteIndexTest to make the close/release threads immediately run rather than async, this avoided race condition issues with validation
| @@ -93,7 +94,9 @@ private boolean isTestType(Class<? extends IPartitioner> klass) | |||
| @Test | |||
| public void byteCompareSerde() | |||
| { | |||
| qt().forAll(AccordGenerators.fromQT(CassandraGenerators.token())).check(token -> { | |||
| // make sure to use simplify as local partitioner can have a type that could generate data too large causing this test to be flakey | |||
| Gen<Token> qt = CassandraGenerators.partitioners().flatMap(p -> CassandraGenerators.token(CassandraGenerators.simplify(p))); | |||
There was a problem hiding this comment.
flakey test during my first CI run. LocalPartitioner can have very complex type, which leads to tokens too large causing the test to be flakey
| @@ -54,7 +54,7 @@ public void partitionerToToken() | |||
| @Test | |||
| public void partitionerKeys() | |||
| { | |||
| qt().forAll(Gens.random(), toGen(CassandraGenerators.partitioners())) | |||
| qt().forAll(Gens.random(), toGen(CassandraGenerators.partitioners().map(CassandraGenerators::simplify))) | |||
There was a problem hiding this comment.
flakey test during my first CI run. LocalPartitioner can have very complex type, which leads to tokens too large causing the test to be flakey
No description provided.