Dynamically set Delta Lake driver memory

[mikix]

Currently, we reserve 4GB of memory for the Delta Lake Java driver stack.

That used to be 1GB and then 2GB and now 4GB, as we dealt with the growing Observation table at BCH.

The hardcoded 4GB is overkill for sites (or even individual tasks) with smaller tables. And updating the hardcoded number for all cases whenever BCH hits a new limit seems bad.

So ideally the ETL would inspect the target table and we'd choose a memory size based on that. Nice and flexible for all cases.