Code

Feature/tiered caching#11

Code

Draft

peteralfonsi wants to merge 134 commits intodummy-tiered-caching-basepeteralfonsi/OpenSearch:dummy-tiered-caching-basefrom feature/tiered-cachingpeteralfonsi/OpenSearch:feature/tiered-cachingCopy head branch name to clipboard

+4,870-320

Commits

Commits on Sep 28, 2023

• Implemented and tested time took for QuerySearchResult objects for use in tiered caching

  

  Peter Alfonsi

  committed
  fcd7476

  Copy full SHA for fcd7476

Commits on Oct 2, 2023

• Moved setting to QueryPhase.execute()

  

  Peter Alfonsi

  committed
  87cbe2a

  Copy full SHA for 87cbe2a

• Moved setting to QueryPhase.execute()

  

  Peter Alfonsi

  committed
  cf18eb9

  Copy full SHA for cf18eb9

Commits on Oct 3, 2023

• spotlessCheck

  Show description for c876de8
  

  Peter Alfonsi

  committed
  c876de8

  Copy full SHA for c876de8

Commits on Oct 4, 2023

• Added first draft of caching policy setup

  

  Peter Alfonsi

  committed
  93a16d4

  Copy full SHA for 93a16d4

Commits on Oct 9, 2023

• Tiered caching framework draft changes

  sgup432

  committed
  8d6c8c4

  Copy full SHA for 8d6c8c4

Commits on Oct 10, 2023

• initial commit

  

  Peter Alfonsi

  committed
  83590e1

  Copy full SHA for 83590e1

• fixed permissions

  

  Peter Alfonsi

  committed
  aa0a5d7

  Copy full SHA for aa0a5d7

• Set up event listener to create RemovalNotification objects, also misc functions

  

  Peter Alfonsi

  committed
  41dc035

  Copy full SHA for 41dc035

Commits on Oct 11, 2023

• Added misc getter methods

  

  Peter Alfonsi

  committed
  4ba49af

  Copy full SHA for 4ba49af

• thread pool stuff

  

  Peter Alfonsi

  committed
  dcb9fad

  Copy full SHA for dcb9fad

• cleanup

  

  Peter Alfonsi

  committed
  b912527

  Copy full SHA for b912527

• Removed partial cache files from disk

  

  Peter Alfonsi

  committed
  44d5e61

  Copy full SHA for 44d5e61

• fixed double counting of on heap hit count

  

  Peter Alfonsi

  committed
  43e2d1d

  Copy full SHA for 43e2d1d

• first attempt at fixing cache closing issue in tests

  

  Peter Alfonsi

  committed
  aa2d28d

  Copy full SHA for aa2d28d

Commits on Oct 12, 2023

• misc stuff for testing

  

  Peter Alfonsi

  committed
  354c311

  Copy full SHA for 354c311

Commits on Oct 13, 2023

• Moved more stuff around for debugging issues

  

  Peter Alfonsi

  committed
  66915b6

  Copy full SHA for 66915b6

• fixed various serialization issues relating to silent ehcache put() and get() failures

  

  Peter Alfonsi

  committed
  f985e63

  Copy full SHA for f985e63

Commits on Oct 16, 2023

• Cleanup

  

  Peter Alfonsi

  committed
  ae300e9

  Copy full SHA for ae300e9

• More cleanup

  

  Peter Alfonsi

  committed
  7a7b41f

  Copy full SHA for 7a7b41f

• Merged in Sagar's serialization changes

  

  Peter Alfonsi

  committed
  ec729d0

  Copy full SHA for ec729d0

• Part 1 of modifying ehcache to take serializable Key

  

  Peter Alfonsi

  committed
  990ba9c

  Copy full SHA for 990ba9c

Commits on Oct 17, 2023

• final attempts at doing it generally before learning that wasnt even needed

  

  Peter Alfonsi

  committed
  d21af15

  Copy full SHA for d21af15

• Changed Ehcache disk tier implementation to not be generic, fixed persistent cache locking issue

  

  Peter Alfonsi

  committed
  d152478

  Copy full SHA for d152478

• Fixed multiple disk tier instance issue, made spillover test pass

  

  Peter Alfonsi

  committed
  2d200fc

  Copy full SHA for 2d200fc

• Removed persistence, cleaned up

  

  Peter Alfonsi

  committed
  8f65db4

  Copy full SHA for 8f65db4

• cleanup

  

  Peter Alfonsi

  committed
  7155afd

  Copy full SHA for 7155afd

Commits on Oct 18, 2023

• added test entries metric for on heap

  

  Peter Alfonsi

  committed
  c760594

  Copy full SHA for c760594

• Fixed issue where multiple mock nodes on one JVM (in IT tests) would cause initialization errors in the cache manager

  

  Peter Alfonsi

  committed
  8510637

  Copy full SHA for 8510637

• Merge branch 'framework-serialized' into update_node_stats

  Show description for 0b2e506
  

  Peter Alfonsi

  committed
  0b2e506

  Copy full SHA for 0b2e506

• Added check for new entries field in IT test, added permissions allowing the domain to run

  

  Peter Alfonsi

  committed
  4680ea7

  Copy full SHA for 4680ea7

Commits on Oct 20, 2023

• Initial implementation for tiered node request cache stats

  

  Peter Alfonsi

  committed
  c300d8b

  Copy full SHA for c300d8b

• Added version checks to streaming functions

  

  Peter Alfonsi

  committed
  9d8d433

  Copy full SHA for 9d8d433

Commits on Oct 23, 2023

• Added UT for RequestCacheStats

  

  Peter Alfonsi

  committed
  7d03562

  Copy full SHA for 7d03562

• Added IT for disk tier stats, plus spotlessApply

  

  Peter Alfonsi

  committed
  c5099f1

  Copy full SHA for c5099f1

• cleaned up

  

  Peter Alfonsi

  committed
  64b4bb3

  Copy full SHA for 64b4bb3

• Implements a memory-efficient roaring bitmap-based keystore for use in disk cache tier

  Show description for 8d13a17
  

  Peter Alfonsi

  committed
  8d13a17

  Copy full SHA for 8d13a17

• Adds and tests took time for QuerySearchResult

  Show description for 08a4f6d
  

  Peter Alfonsi

  committed
  08a4f6d

  Copy full SHA for 08a4f6d

• Addressed Ankit's comments

  Show description for dfd9128
  

  Peter Alfonsi

  committed
  dfd9128

  Copy full SHA for dfd9128

• Removed minimum version variable

  Show description for a9ab327
  

  Peter Alfonsi

  committed
  a9ab327

  Copy full SHA for a9ab327

Commits on Oct 26, 2023

• Changed type for tookTime from long to Long, to address Sagar's comment

  Show description for 4e57f4c
  

  Peter Alfonsi

  committed
  4e57f4c

  Copy full SHA for 4e57f4c

Commits on Oct 31, 2023

• Addressed dblock's comment

  Show description for 1d47f38
  

  Peter Alfonsi

  committed
  1d47f38

  Copy full SHA for 1d47f38

Commits on Nov 2, 2023

• Adds Serializer interface for use in ehcache disk tier and elsewhere; modifies existing disk tier impl to use it in a generic way

  Show description for e39514b
  

  Peter Alfonsi

  committed
  e39514b

  Copy full SHA for e39514b

• Addressed Sagar's comments besides new counter/arraylist setup for removing keys

  Show description for d45e6cf
  

  Peter Alfonsi

  committed
  d45e6cf

  Copy full SHA for d45e6cf

Commits on Nov 6, 2023

• Addressed comments besides moving IT tests into IndicesRequestCacheIT.java

  Show description for 157ca6e
  

  Peter Alfonsi

  committed
  157ca6e

  Copy full SHA for 157ca6e

• Implemented/tested counter+removal list setup to allow more removals

  Show description for 4045616
  

  Peter Alfonsi

  committed
  4045616

  Copy full SHA for 4045616

• Merge branch 'keystore-squashed' into feature/tiered-caching

  Show description for 0825316
  

  Peter Alfonsi

  committed
  0825316

  Copy full SHA for 0825316

• Integrated keystore

  

  Peter Alfonsi

  committed
  8e51749

  Copy full SHA for 8e51749

• removed exceptions from interface signatures

  Show description for 5099968
  

  Peter Alfonsi

  committed
  5099968

  Copy full SHA for 5099968

• Merge branch 'tooktime-squashed' into feature/tiered-caching

  

  Peter Alfonsi

  committed
  207ca99

  Copy full SHA for 207ca99

• Adds caching policy

  Show description for 21dab6f
  

  Peter Alfonsi

  committed
  21dab6f

  Copy full SHA for 21dab6f

• Merge branch 'caching-policy-squashed' into feature/tiered-caching

  Show description for e5cadb0
  

  Peter Alfonsi

  committed
  e5cadb0

  Copy full SHA for e5cadb0

Commits on Nov 7, 2023

• Added first-draft integration for caching policy which will be changed when serializer is merged to make it generic

  Show description for f149eec
  

  Peter Alfonsi

  committed
  f149eec

  Copy full SHA for f149eec

• Added unit test for ehcache tier's took time policy integration

  Show description for ee7301a
  

  Peter Alfonsi

  committed
  ee7301a

  Copy full SHA for ee7301a

Commits on Nov 8, 2023

• Reworked caching policy to sit at tiered cache service level and use V -> W generic transformation function to inspect cache values

  Show description for ee11ed8
  

  Peter Alfonsi

  committed
  ee11ed8

  Copy full SHA for ee11ed8

• Fixed issue with eviction test for IRC tests

  Show description for 5123142
  

  Peter Alfonsi

  committed
  5123142

  Copy full SHA for 5123142

• first parts of cachevalue rework (nonfunctional)

  Show description for 4865827
  

  Peter Alfonsi

  committed
  4865827

  Copy full SHA for 4865827

• Merge commit '157ca6e9557cc5864c45c930be97efa6f17ea269' into cache-value

  Show description for 695521f
  

  Peter Alfonsi

  committed
  695521f

  Copy full SHA for 695521f

Commits on Nov 9, 2023

• Almost finished statsholder impl side, found and deleted a lot of old tiered stuff in indices package

  

  Peter Alfonsi

  committed
  8fb86b7

  Copy full SHA for 8fb86b7

Commits on Nov 10, 2023

• Finished cachevalue rework + disk stats implementation

  Show description for 900230d
  

  Peter Alfonsi

  committed
  900230d

  Copy full SHA for 900230d

Commits on Nov 11, 2023

• Simplified RBM size estimator

  Show description for 07a2e4b
  

  Peter Alfonsi

  committed
  07a2e4b

  Copy full SHA for 07a2e4b

• Merge branch 'keystore-squashed' into feature/tiered-caching

  

  Peter Alfonsi

  committed
  a9df136

  Copy full SHA for a9df136

Commits on Nov 13, 2023

• addressed sagar's comments

  Show description for 2f884c2
  

  Peter Alfonsi

  committed
  2f884c2

  Copy full SHA for 2f884c2

• merged in serializer changes

  Show description for 9d88676
  

  Peter Alfonsi

  committed
  9d88676

  Copy full SHA for 9d88676

• Fixed broken test

  Show description for 4e8ba0b
  

  Peter Alfonsi

  committed
  4e8ba0b

  Copy full SHA for 4e8ba0b

• Merged in cache value changes

  Show description for 41ae53e
  

  Peter Alfonsi

  committed
  41ae53e

  Copy full SHA for 41ae53e

• Re-added and tested serializer for IRC key

  Show description for 09748eb
  

  Peter Alfonsi

  committed
  09748eb

  Copy full SHA for 09748eb

Commits on Nov 14, 2023

• added null check to BR deserializer

  Show description for 8cb83d7
  

  Peter Alfonsi

  committed
  8cb83d7

  Copy full SHA for 8cb83d7

• Fixed ehcache init issue in test cases

  Show description for 111c4e6
  

  Peter Alfonsi

  committed
  111c4e6

  Copy full SHA for 111c4e6

• cleanup, fixed commented line

  Show description for 55b09fe
  

  Peter Alfonsi

  committed
  55b09fe

  Copy full SHA for 55b09fe

• Fixed issues with closing disk tier in tests, added IT for disk specific stats

  Show description for 24a3cb9
  

  Peter Alfonsi

  committed
  24a3cb9

  Copy full SHA for 24a3cb9

• removed print statement

  Show description for 6a280c6
  

  Peter Alfonsi

  committed
  6a280c6

  Copy full SHA for 6a280c6

Commits on Nov 16, 2023

• Removed incorrect line from test

  Show description for 175c5c5
  

  Peter Alfonsi

  committed
  175c5c5

  Copy full SHA for 175c5c5

Commits on Nov 17, 2023

• added required javadoc

  Show description for af5178f
  

  Peter Alfonsi

  committed
  af5178f

  Copy full SHA for af5178f

Commits on Nov 22, 2023

• Optimized modulo calculation with bitmask

  Show description for 7d1ce12
  

  Peter Alfonsi

  committed
  7d1ce12

  Copy full SHA for 7d1ce12

• Addressed Sagar's comments on caching policy/generics

  Show description for 57e948f
  

  Peter Alfonsi

  committed
  57e948f

  Copy full SHA for 57e948f

• Addressed Sagar's other comments

  Show description for c51e0f3
  

  Peter Alfonsi

  committed
  c51e0f3

  Copy full SHA for c51e0f3

• removed dead code

  Show description for 100ca7b
  

  Peter Alfonsi

  committed
  100ca7b

  Copy full SHA for 100ca7b

• removed more dead code

  Show description for c415ec8
  

  Peter Alfonsi

  committed
  c415ec8

  Copy full SHA for c415ec8

Commits on Nov 27, 2023

• Changed memory size estimator to reflect optimized modulo

  Show description for 93fa4f8
  

  Peter Alfonsi

  committed
  93fa4f8

  Copy full SHA for 93fa4f8

• Optimizes modulo using bitmask

  

  Peter Alfonsi

  committed
  433470e

  Copy full SHA for 433470e

Commits on Nov 28, 2023

• Update IndicesRequestCache.java

  Show description for 52b9dfd
  kiranprakash154

  committed
  52b9dfd

  Copy full SHA for 52b9dfd

• More updates to disk cache cleanup

  kiranprakash154

  committed
  5bab65d

  Copy full SHA for 5bab65d

Commits on Nov 29, 2023

• register INDICES_REQUEST_CACHE_DISK_CLEAN_THRESHOLD_SETTING in clustersettings.java

  kiranprakash154

  committed
  4500090

  Copy full SHA for 4500090

• make INDICES_REQUEST_CACHE_DISK_CLEAN_INTERVAL_SETTING dynamic property

  kiranprakash154

  committed
  4199bc2

  Copy full SHA for 4199bc2

• minor changes

  kiranprakash154

  committed
  6e05b9d

  Copy full SHA for 6e05b9d

• change the way we calculate disk key staleness

  kiranprakash154

  committed
  aa3c39c

  Copy full SHA for aa3c39c

• fix breaking tests

  kiranprakash154

  committed
  83d70f7

  Copy full SHA for 83d70f7

Commits on Nov 30, 2023

• dummy commit

  

  Peter Alfonsi

  committed
  e41321d

  Copy full SHA for e41321d

• Changed github action to run on my fork. REVERT THISgit add .

  

  Peter Alfonsi

  committed
  a51da38

  Copy full SHA for a51da38

• added broken javadoc

  

  Peter Alfonsi

  committed
  f0a26b9

  Copy full SHA for f0a26b9

• Fix test to include IndexShard

  kiranprakash154

  committed
  caae291

  Copy full SHA for caae291

• UT for testing invalidate of DiskTier is called

  kiranprakash154

  committed
  9d0eab1

  Copy full SHA for 9d0eab1

• added upload artifact section to workflow, spotlessApply

  

  Peter Alfonsi

  committed
  01bcad7

  Copy full SHA for 01bcad7

Commits on Dec 1, 2023

• Added notification for REMOVED case in ehcache event listener

  

  Peter Alfonsi

  committed
  c6a93a4

  Copy full SHA for c6a93a4

Commits on Dec 4, 2023

• draft changes for enable disk tier setting

  

  Peter Alfonsi

  committed
  93ef124

  Copy full SHA for 93ef124

• reverted modified github actions

  

  Peter Alfonsi

  committed
  20114ca

  Copy full SHA for 20114ca

• reverted modified github actions

  

  Peter Alfonsi

  committed
  b7fcac7

  Copy full SHA for b7fcac7

Commits on Dec 5, 2023

• Introduce CleanupStatus to keysToClean

  kiranprakash154

  committed
  5c472ae

  Copy full SHA for 5c472ae

• use that cleanupStatus and logic to update staleKeyEntries

  kiranprakash154

  committed
  be10a16

  Copy full SHA for be10a16

• register INDICES_REQUEST_CACHE_DISK_CLEAN_INTERVAL_SETTING to cluster settings

  kiranprakash154

  committed
  4c8b240

  Copy full SHA for 4c8b240

• Add removal listener to update eh cache stats

  kiranprakash154

  committed
  eeb973e

  Copy full SHA for eeb973e

• re-organize imports

  kiranprakash154

  committed
  f22cdaf

  Copy full SHA for f22cdaf

• Add IT tests

  kiranprakash154

  committed
  280cb07

  Copy full SHA for 280cb07

• Merge branch 'feature/tiered-caching' into kp/delete-stale-keys

  Show description for 27f122b
  kiranprakash154

  authored
  27f122b

  Copy full SHA for 27f122b

• Added RBM size setting + UT

  

  Peter Alfonsi

  committed
  295ff1a

  Copy full SHA for 295ff1a

• added UT for setting switch

  

  Peter Alfonsi

  committed
  163d7fe

  Copy full SHA for 163d7fe

• added skeleton for stale key threshold setting

  

  Peter Alfonsi

  committed
  29d31b4

  Copy full SHA for 29d31b4

• Added commented-out lines to opensearch.yml

  

  Peter Alfonsi

  committed
  856160e

  Copy full SHA for 856160e

Commits on Dec 6, 2023

• added feature flag

  

  Peter Alfonsi

  committed
  4bdf459

  Copy full SHA for 4bdf459

Commits on Dec 7, 2023

• added ehcache cluster settings

  

  Peter Alfonsi

  committed
  347a14c

  Copy full SHA for 347a14c

• added feature flag setting to disk tier IT

  

  Peter Alfonsi

  committed
  64c339a

  Copy full SHA for 64c339a

• added null checks to key and dummy string serializers

  

  Peter Alfonsi

  committed
  90c0f9a

  Copy full SHA for 90c0f9a

Commits on Dec 8, 2023

• Integrated misc changes from other branch

  

  Peter Alfonsi

  committed
  47453e1

  Copy full SHA for 47453e1

Commits on Dec 11, 2023

• changed dummy disk cache size from 1 MB -> 10 GB

  

  Peter Alfonsi

  committed
  216d99d

  Copy full SHA for 216d99d

• Merge branch 'misc-settings' into enable-disk-setting

  

  Peter Alfonsi

  committed
  6483c69

  Copy full SHA for 6483c69

Commits on Dec 12, 2023

• added first draft of dynamic disk tier enable/disable logic

  

  Peter Alfonsi

  committed
  8db79e0

  Copy full SHA for 8db79e0

• added IT testing toggling disk tier dynamic setting

  Show description for 153c575
  

  Peter Alfonsi

  committed
  153c575

  Copy full SHA for 153c575

• cleanup

  Show description for 52dad7a
  

  Peter Alfonsi

  committed
  52dad7a

  Copy full SHA for 52dad7a

• removed disk tier checks from IT where disk tier is not turned on

  

  Peter Alfonsi

  committed
  1671e2f

  Copy full SHA for 1671e2f

• added checks to disk enable/disable fns, condensed IT

  

  Peter Alfonsi

  committed
  24b5f14

  Copy full SHA for 24b5f14

• set default disk tier size to 100 mb

  Show description for 1c1a2b5
  

  Peter Alfonsi

  committed
  1c1a2b5

  Copy full SHA for 1c1a2b5

Commits on Dec 13, 2023

• invalidateAll() also invalidates keys from inactive disk tier

  Show description for 2758ee3
  

  Peter Alfonsi

  committed
  2758ee3

  Copy full SHA for 2758ee3

• set disk threshold policy to 0

  

  Peter Alfonsi

  committed
  f8b0425

  Copy full SHA for f8b0425

Commits on Dec 15, 2023

• refactor cleanCache & cleanDiskCache to share methods

  kiranprakash154

  committed
  8f0b5bc

  Copy full SHA for 8f0b5bc

• null checks for indexShard

  kiranprakash154

  committed
  8c996b3

  Copy full SHA for 8c996b3

• some bugs i found

  kiranprakash154

  committed
  4bd1503

  Copy full SHA for 4bd1503

• use assertNumCacheEntries

  kiranprakash154

  committed
  1911fa6

  Copy full SHA for 1911fa6

• update print statement

  kiranprakash154

  committed
  ae0b471

  Copy full SHA for ae0b471

• edit sleep time

  kiranprakash154

  committed
  fce110d

  Copy full SHA for fce110d

• changed setting paths from index to indices for non-index level settings

  

  Peter Alfonsi

  committed
  7d2ea2e

  Copy full SHA for 7d2ea2e

Commits on Dec 18, 2023

• Addressed Kiran's comments

  Show description for 49c9bbd
  

  Peter Alfonsi

  committed
  49c9bbd

  Copy full SHA for 49c9bbd

• Merge branch 'misc-settings' into feature/tiered-caching

  

  Peter Alfonsi

  committed
  dc3fd1e

  Copy full SHA for dc3fd1e

Commits on Dec 19, 2023

• Merge branch 'kp/delete-stale-keys' into feature/tiered-caching

  

  Peter Alfonsi

  committed
  f375874

  Copy full SHA for f375874