Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
# Motivation
- Allow users to implement their own object storage plugins without
modifying this repository.
- Prepare the import existing object from the object storage feature by
dissociating Result metadata identifier from the key used in the object
storage to reference the actual data.
# Description
- Move object storage interfaces from ArmoniK.Core.Common to
ArmoniK.Core.Base package so that they can be implemented outside of
ArmoniK.Core.
- Change the Result data schema in database to add an OpaqueId which
represent the identifier used by the ObjectStorage to refer to the
Result data.
- Dynamically load object storages as done for the queue. It works on
the same principle : we use look for an implementation of
`IDependencyInjectionBuildable` and instanciate it. Then, this interface
is responsible for adding implementation of `IObjectStorage` to its
given `ServicesCollection`.
- The AddQueue function has been generalised to support ObjectStorages
and QueueStorages. It should also be able to support database and
partition plugins loading when it will be available.
- Update ArmoniK.Core infrastructure to properly use the dynamically
loaded object storages.
- Implement Embed object storage that stores the data into the key,
therefore inserting them completely in the database.
- Adds documentation to explain how to migrate an existing ArmoniK
database to the new definition of Result.
# Testing
- Unit tests and integration tests were executed locally and in the
pipeline.
- Database migration was tested with the following procedure:
First checkout and deploy with the previous release from ArmoniK.Core:
```bash
git checkout 0.28.0
just object=redis worker=htcmock tag=0.28.0 deploy
```
Then execute a job with a few tasks so that there is data to migrate in
the database. Do not forget to keep data in the Object Storage after the
execution (disable Purge from the session) to fully test that the
association will be done properly by the adapter.
```bash
docker run --net armonik_network --rm \
-e HtcMock__NTasks=10 \
-e HtcMock__TotalCalculationTime=00:00:00.100 \
-e HtcMock__DataSize=1 \
-e HtcMock__MemorySize=1 \
-e HtcMock__PurgeData=false \
-e HtcMock__Partition=TestPartition0 \
-e GrpcClient__Endpoint=http://armonik.control.submitter:1080 \
dockerhubaneo/armonik_core_htcmock_test_client:0.28.0
```
Once the job is finished, go to the version of ArmoniK.Core on this
branch and deploy it.
```bash
git checkout jg/dynamicobject
just object=redis worker=htcmock build-deploy
```
Connect to the database and execute the migration query found in the
documentation added by this PR.
```bash
docker exec -it database mongosh database
```
Download results metadata and try to access its data from the object
storage. The two requests should work without issue. Keep in mind that
you should replace the result_id and the session_id.
```bash
docker run --net armonik_network fullstorydev/grpcurl -plaintext -d '{"result_id" : "80e8c67d-3093-48f4-a635-c1cc10299bce" }' armonik.control.submitter:1080 armonik.api.grpc.v1.results.Results/GetResult
docker run --net armonik_network fullstorydev/grpcurl -plaintext -d '{"result_id" : "80e8c67d-3093-48f4-a635-c1cc10299bce", "session_id" : "3df359ad-e69e-426a-970f-11e798e9b1ae" }' armonik.control.submitter:1080 armonik.api.grpc.v1.results.Results/DownloadResultData
```
# Impact
- Performances with the existing object storage plugins should not be
impacted.
- Embed object storage should be more performant since it will store
data in the database but will increase the memory usage of the database.
Be careful for large data as MongoDB cannot store documents larger than
16Mb.
- Existing databases should be migrated according to the suggested
procedure.
- Object Storage configuration will have to be changed according to the
object storage used (see the outputs of our modules).
# Additional Information
- Using a string for OpaqueId instead of binary data during the
migration does not work properly. String cannot be automatically
deserialized into a byte array.
Some links I studied to complete this PR.
-
https://stackoverflow.com/questions/61794277/method-does-not-have-an-implementation-when-loading-assemblies-into-a-new-assemb
- https://www.mongodb.com/docs/current/reference/method/BinData/
-
https://www.mongodb.com/docs/manual/reference/operator/aggregation/convert/
- https://www.mongodb.com/docs/manual/reference/method/HexData/
-
https://www.mongodb.com/docs/manual/reference/operator/aggregation/function/
# Checklist
- [x] My code adheres to the coding and style guidelines of the project.
- [x] I have performed a self-review of my code.
- [x] I have commented my code, particularly in hard-to-understand
areas.
- [x] I have made corresponding changes to the documentation.
- [x] I have thoroughly tested my modifications and added tests when
necessary.
- [x] Tests pass locally and in the CI.
- [x] I have assessed the performance impact of my modifications.- Loading branch information
