Improvements for reference integrity checking for large collection inventories (~>100MB)

[al-niessner]

🗒️ Summary

Expand so that validate-refs can use all available memory before failing. Well, 128 GB anyway. Also fixed XPath expression problem.

⚙️ Test Data and/or Report

All automated unit tests pass below

♻️ Related Issues

Closes #748
Closes #750

[jordanpadams]

@al-niessner is there a way we can paginate this into smaller chunks to avoid requiring so much memory?

[jordanpadams]

@al-niessner See comment above ☝️

[jordanpadams]

If possible, I would like us to find a way to cut down on the amount of memory required for this type of task. The example that caused the software to break is not going to be the largest data set we have. We need to be able to support much much larger.

[al-niessner]

@jordanpadams

The code is not downloading 1000000 labels at once which would allow for pagination. It loads one lidvid at a time over a loop. It keeps some stuff in local memory but not much.

The actual size error #748 looks like it is trying to get a very, very large collection and HTTP stack in java could not load it. Again, nothing there to paginate.

[jordanpadams]

@al-niessner Sorry for the confusion here. Removing paginate from the question, is there any way we can improve performance from very very large collections where it would not require a huge amount of memory in order to perform this operation?

Would it be more performant to use the API instead?

[al-niessner]

@jordanpadams

Given the restrictions/constraints of the tools selected (spring, etc) it seems that large memory is what you need. Since it takes an int as its max size, 2GB is the largest collection that could ever be managed without a tooling change. This one was 187+ MB.

No, there is really no simpler way. It may be that when you ask the API for the collection, it does not bother to give you the big part saving the download and giving the appearance that the API would do it -- all that magic keyword stuff that is done API where it decides what you need to know for you. Unfortunately, this tool needs the part (list of references) that the API usually decides you do not need. Now, the API might get a bit unhappy if you were to ask it for all the products in the collection.

The increase in buffer is not what Java will allocate. It is what Java may allocate. If you ask for a big item, then give it lots of memory.

[jordanpadams]

@al-niessner hmmmm copy. ok. we may need to figure out how to create a max size for those collection lists (>2GB). Will merge this, and then re-evaluate a new solution.