-
Notifications
You must be signed in to change notification settings - Fork 18
wxslucene
This allows Lucene indexes to be stored within a grid rather than on disk or using a RAMDirectory. This allows large indexes to be stored in memory instead of on disk. The large indexes are transparently stored on a grid of several JVMs. The heaps of the JVMs collectively provide the storage for the large indexes. This data can be replicated for fault tolerance also. It should provide performance levels faster than most disk based systems as practically as fast as SAN based systems. The main advantage is that it provides a memory based alternative to RAMDirectory for large index sizes. It also allows a lucene application to be start without needed to copy an index to a RAMDirectory everytime. Now, the application can just connect to the grid and it’s ready to go immediately.
All jars are prebuilt and available here
Full source code for the Directory in here on github.
Lucene and all its dependencies
- wxslucene.jar
- wxsutils.jar
- ogclient.jar or objectgrid.jar
- wxsutils.jar
- wxslucene.jar
- objectgrid.jar and all its dependencies
The objectgrid.xml and deployment.xml files in the test/resources should be used with appropriate modifications for partition counts and replication strategy. The only file requiring modification is the deployment.xml. The only things here to modify are the partition count as well as replication strategy.
Each directory and its file contents are stored in a separate map with a prefix of ChunkMap.
The Lucene plugin is configured using two property files that should be on the classpath. The wxsutils.properties file is used to connect to a grid and the wxslucene.properties file configures the directory plugin.
See here
Here is an example wxsutils.properties
The wxslucene.properties file must be in a folder on the classpath. All directories within the client share the same configuration.
Here is an example wxslucene.properties
This is a global or per directory property. If set to true then when GridDirectory copies a file based index to the grid, it checks each
destination file to make sure it’s the same as the source file using an MD5 digest. This is usually used to diagnose any odd lucene exceptions like bounds exceptions and so on when running searches. The usual culprit is a corrupt index in the grid. This property allows the grid to be easily verified.
This displays the current hit rate for the local block cache as well as the overall compression ratio for the blocks. This is displayed every 20k block gets from a directory. This can be specified per directory if needed.
This batches writes to a file together so that they are essentially bulk copied to the grid. This improves performance significantly in that multiple blocks are written to each JVM together using a single RPC versus an RPC per block if this property is false.
The compression can significantly reduce the memory required for the index and reduces the network bandwidth required between the lucene JVMs and the grid. There is a CPU cost and this cost needs to be weighed against the advantages. Compression enabled impacts write performance much more than read performance.
If a local block cache is enabled then the cost of compression is significantly reduced as the block cache holds uncompressed blocks. Hence, enabling compression with the block cache is a recommended configuration as its a very good compromise on performance and memory usage.
Index files are broken in to chunks or blocks of a fixed size when stored in the grid. The blockSize parameter specifies how big those blocks are. 16k seems a common value for these kinds of system.
This is the maximum number of blocks to cache in a LRU cache for each directory. The blocks are cached in uncompressed form. Thus means the maximum memory consumption will be block_cache_size * block_size * #directories in use in a JVM. The block cache is only enabled if this property is specified. It should only be used with read only indexes. This cache is not synchronized across the cluster and so is only usable in situations where the index is read only.
The block cache typically makes an enormous performance difference when enabled. It should be enabled if at all possible. It can also mask compression cost as every cache hit returns an uncompressed block.
How big this value should be depends on the indexes but a good starting value is 1024
This compresses any blocks stored in the local block cache. I’d typically expect between 2x and 3x compression. This does mean there is some CPU overhead when fetching blocks from the local cache. This can only be specified for a specific directory if the block_cache_size property is also specified.
Normally the key for each block is a String composed of the “directoryPath/file#NNNN” where NNNN is the block number. This string can be quite long so this property stores the key instead as the md5 hash of this string. This is a 16 byte array and is typically much smaller than the string would be. It is possible that two blocks hash to the same byte[] but this is extremely unlikely. But, if any corrupt indexes are seen after setting this property to true then there has likely been a collision and you cannot turn on this feature as a result.
Many of the properties can be made exclusive to a particular directory. The convention is as follows:
DirectoryName.propertyName=xxx
The DirectoryName is the file path string passed to ClientGridDirectory OR GridDirectory.
The following properties can be specified per directory:
- block_cache_size
- compress_block_cache
- log_hit_rate
- key_as_digest
- verify_copy
For an example, a local block cache can be enabled for the directory “/lucene/index/Country” using the property
/lucene/index/Country.block_cache_size=1024
Generational garbage collection is recommended for read only indexes. We recommend larger 64 bit JVMs typically if required. For example, -Xmx8G. You should specify “-server” on client and container JVMs. When sizing a grid, use a worst case high heap threshold of maybe 80% for example. So, a JVM can be expected to hold 80% of 8G or 6.4Gb of blocks.
Typically with a 30Gb index, thats 60Gb with replication. This means a minimum configuration of 60/6.4 JVMs or ~9.4 or rounded up, 10 × 8GB heap JVMs. A 64GB box should be able to host 7 × 8GB JVMs. So we’d need 2 boxes at a minimum to hold this grid. We’d add one more box with an additional 7 × 8GB JVMs to provide redundancy in the case of a box failing. Remember, you need enough capacity to handle a single box failure as a minimum. If we’re running heaps at 80% and just using 2 boxes then losing one box would likely result in out of memory exceptions on the survivors as the grid tries to recover. So, I’d run 3 boxes so that even if we lose a box, we would be left with the system running at healthy heap level.
WXS is cheaper to license in these scenarios on boxes with more memory as it’s the total number of CPU cores that determines price not memory so buy boxes with more memory.
The partition count is usually 5x the number of JVMs used in a ‘normal’ system after rounding up to the next highest prime number. So, 4 JVMs would result in 20 partitions which isn’t prime so 23 would be the partition count.
Example command line to start a WXS grid container would be:
startOgServer.sh server01 -objectgridFile xml/objectgrid.xml -deploymentPolicyFile xml/deployment.xml -jvmArgs -d64 -Xmx8G -Dcom.sun.management.jmxremote -verbose:gc -cp jars/wxsutils-1.3-SNAPSHOT.jar:jars/wxslucene-1.0-SNAPSHOT.jar
The wxslucene plugin uses JMX to expose statistics metrics. This means it expects an MBeanServer to be started. The command line argument “-Dcom.sun.management.jmxremote” should start a JMX MBeanServer in a JVM. It must be specified on any JVMs using the wxslucene plugin.
You can attach to a client JVM using jconsole to easily see the MBeans for metrics.