RocksDB Merge Operator for Open (MPU) Key Location Update

[ivandika3]

Has the community experimented on the use of the RocksDB Merge Operator for read-modify-write operation (e.g. MPU commit parts)?

[kerneltime]

@ivandika3 MPU commit is a painful operation today if the part size is not managed by the application. It would be interesting to see how we can leverage the merge operator via Java interface. A different alternative would be is to keep the parts as is and only reference them in the committed key. The client can paginate and get all the blocks if the part count is high. This would allow the client to start reading the content while it pre fetches future ranges to be read in.

[ivandika3]

Thank you for the reply. Was thinking whether the RocksDB merge operator can only store the delta in the WAL instead of the whole value (to alleviate the issue #6869), but after some research I don't think this is the case. Merge operator aim seems to be to increase the performance of read-modify-write ops, not reducing WAL space. I think we can consider using this in the future.

A different alternative would be is to keep the parts as is and only reference them in the committed key. The client can paginate and get all the blocks if the part count is high. This would allow the client to start reading the content while it pre fetches future ranges to be read in.

Yes, I think we can change the MPU flow so that the previous parts are not always referred by the subsequent parts. Perhaps we can continue the discussion in https://issues.apache.org/jira/browse/HDDS-8238.