[Fix] Part-1 - LevelDbStore

[cschuchardt88]

Description

This fixes the store's dotnet memory management for GC, a long with adding new feature options, easier Exception catching.

Also exposes the store.

Change Log

• Added more option features to ReadOptions, WriteOptions and Options.
• Added LevelDBHandle.cs for GC and easy management.
• Added IEnumerable<KeyValuePair<byte[], byte[]>> to Store.cs to Iterate over the whole store.
• Changed namespace from Neo.IO.Data.LevelDB to Neo.IO.Storage.LevelDB
• Added MaxOpenFiles=4096, SnappyCompression and FilterPolicy=10 option to Store.
• Added code comments
• Added thread-safety to batching since leveldb doesn't support multi-thread in batching.

Type of change

• Optimization (the change is only an optimization)
• Style (the change is only a code style for better maintenance or standard purpose)
• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

How Has This Been Tested?

Unit tests

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• Any dependent changes have been merged and published in downstream modules

[vncoelho]

can this PR be tested without PART 2?

[cschuchardt88]

can this PR be tested without PART 2?

Yes

[Jim8y]

Hey Chris, can you please add a few UTs to demonstrate how your pr improves the leveldb? I am reviewing your code, but dont really clear how and where the improve works.

[cschuchardt88]

Hey Chris, can you please add a few UTs to demonstrate how your pr improves the leveldb? I am reviewing your code, but dont really clear how and where the improve works.

Part-2 are the tests

[Jim8y]

Will check part 2 as well then, but please avoid doing this, cause it makes a single pr not reviewable.

[cschuchardt88]

Will check part 2 as well then, but please avoid doing this, cause it makes a single pr not reviewable.

Just trying to make it small.

[cschuchardt88]

Added thread-safety to batching since leveldb doesn't support multi-thread in batching.

[Jim8y]

@superboyiii can you please rake a look at this pr and do some tests on it?

[superboyiii]

@superboyiii can you please rake a look at this pr and do some tests on it?

Sure

[superboyiii]

@cschuchardt88 Seems not an issue from this PR because it occurs on master in block 2724677, I will open an issue.

[cschuchardt88]

@superboyiii Is your testing finished for this PR? are we good to move forward?

[superboyiii]

@superboyiii Is your testing finished for this PR? are we good to move forward?

I've tried functional tests and compatiblity tests before, all seems works well. But not sure how to test GC performance. Do you have any good way or suggestion?

[cschuchardt88]

@superboyiii Is your testing finished for this PR? are we good to move forward?

I've tried functional tests and compatiblity tests before, all seems works well. But not sure how to test GC performance. Do you have any good way or suggestion?

GC is an automated thing built into dotnet, that picks up stuff that is unallocated. At least now it handled better. So the GC doesn't removes anything that we are using in unmanaged state. If we were using huge chunks of data, we could measure it. But the data we handle is so small, nothing to measure. It's only Pointers we are talking about.

[superboyiii]

GC is an automated thing built into dotnet, that picks up stuff that is unallocated. At least now it handled better. So the GC doesn't removes anything that we are using in unmanaged state. If we were using huge chunks of data, we could measure it. But the data we handle is so small, nothing to measure. It's only Pointers we are talking about.

Will this improvement make memory increase less when node is running?

[cschuchardt88]

GC is an automated thing built into dotnet, that picks up stuff that is unallocated. At least now it handled better. So the GC doesn't removes anything that we are using in unmanaged state. If we were using huge chunks of data, we could measure it. But the data we handle is so small, nothing to measure. It's only Pointers we are talking about.

Will this improvement make memory increase less when node is running?

No memory will be the same. Just won't get un-allocated randomly by GC. In other words the GC knows what it is.

Can pick in Performance Profiler In Visual Studio 2022

[superboyiii]

Tested with #3415, seems works well and data is compatible. Memory usage is stable.

[shargon]

what was the previous value?

[cschuchardt88]

CreateIfMissing = true,
FilterPolicy = Native.leveldb_filterpolicy_create_bloom(15)

CompressionLevel is enabled by default in leveldb

[shargon]

@Jim8y already introduced something to ensure that is not used in non-thread safe cases

[cschuchardt88]

Still best to do this. All of leveldb is thread safe, except writebatch, isn't thread safe. Better to do it at the source. You can use leveldb in multi-threads. Just not writebatch.