Add hardware-versioned quota based on Flavor extra_specs

[joker-at-work]

We want to support adding cores/ram quota for different hardware versions. Reading the quota:hw_version property from all flavors' extra_specs, we build a list of supported hardware versions and create quota resources of the form cores_{hw_version}.

Change-Id: Ic7725f49a788188e3b1f046ed857c582b2646819

[fwiesel]

I like the approach, but I am not too happy with the function signatures having to change, and that the changes are spread over an upstream class.

It is the result of a decision we have taken before your change, so it is more a suggestion from my side, than a change I expect you to implement:

I am wondering, if it wouldn't make sense to create a derived SAPQuotaEngine like this:

class QuotaEngine(object):
    def __init__(self, resources=None):
      self._resources = resources or {}

    def do_something_with_resources(self):
      print(self._resources)


class SAPQuotaEngine(QuotaEngine):
    def __init__(self, resources=None):
      super().__init__(resources)

    @property
    def _resources(self):
      resources = self._original_resources.copy()
      # Add our custom resources logic here
      resources["Added"] = "MyStuff"
      return resources

    @_resources.setter
    def _resources(self, resources):
      self._original_resources = resources


resources = {'Normal': '1'}

a = QuotaEngine(resources)
a.do_something_with_resources()

b = SAPQuotaEngine(resources)
b.do_something_with_resources()

[joker-at-work]

I am wondering, if it wouldn't make sense to create a derived SAPQuotaEngine like this:

Yes, that would probably make more sense than changing everyone to use self.resources instead of self._resources. That's the approac Cinder does for implementing the VolumeTypeQuotaEngine. I refrained from that at time, iirc, because initializing the quota-engine into QUOTAS is hard-coded - but that would be a single line change then, which is much easier to port. Great suggestion.

[joker-at-work]

Updated with SAPQuotaEngine, caching and also introduced SAPCustomResource for another place in the code.

[fwiesel]

If we use the proper caching infrastructure, wouldn't it make sense to use the configured default one (memcached in our case)?

The advantage would be that it would be more quickly consistent.

[joker-at-work]

I wasn't sure it's worth the serialization/deserialization of CountableResource objects and references to functions - is that even supported with memcached as backend?

[fwiesel]

I missed the part that it stores a function, so I needed to check it first myself.
AFAIK, the caching uses pickling to store the data, and that handles even

functions (built-in and user-defined) accessible from the top level of a module (using def, not lambda);

from nova import quota
import pickle

[pickle.dumps(res) for res in quota.QUOTAS._resources.values()]

So, we might be fine here using a different caching backend.

[fwiesel]

If we should decide to use an external cache, then I think it would be better to remove this invalidation. The cache may very well be still valid then.

[majewsky]

My side of this is in sapcc/limes#488.

As already discussed with Johannes, I would like to leave the door open for having the instances quota also be hardware-versioned. Since instances_$VERSION conflicts with instances_$FLAVOR, I have based my PR on the alternate naming pattern of hw_version_$VERSION_{cores,instances,ram} that we discussed. Would that be acceptable for everyone?

[joker-at-work]

Updated to use the prefix instead of the postfix for hardware versions.