uptake hive 4.x beta version

[murali-shris]

Is your feature request related to a problem? Please describe.

hive older version has few issues related to box corruption, updating hive box in isolate.

Describe the solution you'd like

uptake changes in persistence for hive major version and check for performance improvements.

Describe alternatives you've considered

No response

Additional context

No response

[murali-shris]

Progress made on hive 4.x uptake so far

1. Ran into an issue while running dart sample code to open a box
   IsarNotReadyError: Could not initialize IsarCore library for processor architecture "linux_x64". If you create a Flutter app, make sure to add isar_flutter_libs to your dependencies. For Dart-only apps or unit tests, make sure to place the correct Isar binary in the correct directory.

Invalid argument(s): Failed to load dynamic library 'libisar.so': libisar.so: cannot open shared object file: No such file or directory
#0      initializePlatformBindings (package:isar/src/native/native.dart:26:5)
#1      IsarCore._initialize (package:isar/src/isar_core.dart:42:20)
#2      new _IsarImpl.open (package:isar/src/impl/isar_impl.dart:50:14)
#3      Isar.open (package:isar/src/isar.dart:88:28)
#4      Hive.box (package:hive/src/hive.dart:80:23)
#5      main (file:///home/murali/git/at_protocol/at_server/packages/at_persistence_secondary_server/test/hive_new_test.dart:6:18)
#6      _delayEntrypointInvocation.<anonymous closure> (dart:isolate-patch/isolate_patch.dart:297:19)
#7      _RawReceivePort._handleMessage (dart:isolate-patch/isolate_patch.dart:184:12)

Solution:
Download isar library from https://github.com/isar/isar/releases/tag/4.0.0-dev.14
and add this line to sample/unit test

Isar.initialize(<path to isar lib>);
e.g in macos
Isar.initialize('libisar_macos.dylib')

2. generateSecureKey is no longer part of Hive. We use this method to generate a key and encrypt the box. New Hive allows user to pass encryptionKey in String type. There is no explanation in dart doc what type of encryption key can be passed. whether it allows any random string to be used as encryption key ? I have to test this.
3. Type adapters are no longer part of new Hive. So it is now easier to store and retrieve custom class object. The custom class should have toJson and fromJson impl.
4. Old Hive.add(..) returns the auto increment key which use as commit log id. New Hive no longer returns auto increment key and doesn't expose any method to retrieve the auto increment key after adding. Commit log logic has to be modified. We have to maintain our own key to maintain latest commit id.
5. Analysed whether our existing hive box files can be opened with new Hive methods. Not possible since Hive uses .isar extension for new boxes. Hence we have to come up with custom logic to migrate old box data to new box. Will try this migration for these scenarios before I proceed with other code changes for commit log.

• data migration with simple key-value pair without encryption/type adapter
• data migration with simple key-value pair plus encryption
• data migration with custom class and encryption?
  @gkc @cpswan @cconstab

[murali-shris]

Raised an issue in hive repo for migrating hive box to latest version
isar/hive#1298

[murali-shris]

I will resume working on 4x changes in persistence and pass the branch to @purnimavenkatasubbu @VJag for benchmarking performance. Based on the outcome of benchmarking, we will decide the priority of migrating old hive box to new one.

[murali-shris]

Code changes done in persistence.Currently fixing unit tests in persistence

[murali-shris]

• Fixed unit tests in persistence
• ran unit tests with persistence changes in server and fixed. All tests are passing when run individually. When run using dart run test command, tests are failing. Have to investigate
• @purnimavenkatasubbu Ran the server in local with new changes and has reported few issues. Currently investigating and fixing them.
  Plan for upcoming sprint
• Fix pending issues and run performance tests

[purnimavenkatasubbu]

Performance testing summary performance testing summary

[murali-shris]

hive v4 is not actively maintained.
isar/hive#1225 (comment)
Explore https://pub.dev/packages/hive_ce pointed out by @XavierChanth

[murali-shris]

Uptaking hive_ce was straightforward. Works fine without any code changes.
Tried to auto generated adapters as per documentation in. https://pub.dev/packages/hive_ce
The problem is order of metadata changes in auto generated adapter because of which there is an error reading data from old hive boxes. So left the adapter code as it is.
#2162

[murali-shris]

@purnimavenkatasubbu Please run some performance tests with trunk branch and test_hive_ce branch of server.

[XavierChanth]

That's good to hear, let's hope the performance tests are favourable

[Rexios80]

@murali-shris

Hello! I'm the maintainer of Hive CE.

The problem is order of metadata changes in auto generated adapter because of which there is an error reading data from old hive boxes. So left the adapter code as it is.

Did you try the migration tool mentioned in those docs? Following those instructions should result in generating type adapters that are identical to the ones you currently have.

https://pub.dev/packages/hive_ce#migrating-to-generateadapters

[murali-shris]

@murali-shris

Hello! I'm the maintainer of Hive CE.

The problem is order of metadata changes in auto generated adapter because of which there is an error reading data from old hive boxes. So left the adapter code as it is.

Did you try the migration tool mentioned in those docs? Following those instructions should result in generating type adapters that are identical to the ones you currently have.

https://pub.dev/packages/hive_ce#migrating-to-generateadapters

@Rexios80 Yeah. I have followed the document link already and generated the adapters. There is no issue in hive_ce generating the adapter code. Problem was the way we have implemented the adapters from our side. Order of attributes declaration and order of writing/reading in adapters do not match. I will re-order the attributes in our code and give it a try again. Thanks a lot for the help.

[Rexios80]

Order of attributes declaration and order of writing/reading in adapters do not match.

The migration tool takes this into account. The existing indices defined in your HiveField annotations are maintained. The actual order of reads/writes in the adapter code may have changed, but as long as the indices are the same it shouldn't change any behavior.

Manually reordering the fields instead of using the migration tool will not handle gaps in indices caused by removed fields.

[murali-shris]

Will recheck the migration tool based on the feedback by @Rexios80

[murali-shris]

Will recheck the migration tool based on the feedback by @Rexios80

@Rexios80
In the migration steps, I am able to run successfully until step 3. All the adapters got generated with the extension *.g.dart
4. Revert the build.yaml changes
5. Remove all explicit HiveType and HiveField annotations from your model classes
6. Run the build_runner again
In step 6 is this the correct command ?
dart pub run build_runner build --delete-conflicting-outputs
After step 6 all the generated files *.g.dart got deleted

[Rexios80]

That is expected since you haven't removed the old annotations yet in step 3. That is also why you have to generate twice.

Yes that command works but technically it starts with dart run since dart pub run is deprecated.