Skip to content

Commit

Permalink
Generic targeted refresh support
Browse files Browse the repository at this point in the history
Updated the ems_refresher_mixin to support a pattern for targeted refresh.  This
was largely taken from the VMWare targeted refresh process.

The major difference with the targeted refresh approach is that refreshers
should collect inventory from the provider separate from parsing that inventory.
In many more recent refreshers, the inventory is collected as it's being parsed.
The problem this poses with targeted refresh is that it doesn't allow the
refresher to filter the data down to the specific targets that should be
refreshed.

This implementation allows "legacy" refreshers (those that don't yet support
targeted refresh) to continue to collect inventory while parsing it.  However,
as providers implement targeted refresh, they should adopt the process of
separately collecting inventory.

The other major change with this implementation is that now the VMWare refresher
uses the same ems_refersher_mixin that all other refreshers use.  This helps to
solidify the ability for the mixin to handle both non-targeted and targeted
refreshes.

The major caveat to this approach is that the logging from the old VMWare
refresher is slightly different from the ems_refresher_mixin.  If log analysis
tools exist that rely on the old logging, they will need to be updated to
examine the new logging labels (see updated Benchmark calls in this commit for
details).
  • Loading branch information
blomquisg committed Mar 31, 2016
1 parent f6cb972 commit 4050485
Show file tree
Hide file tree
Showing 17 changed files with 160 additions and 174 deletions.
106 changes: 86 additions & 20 deletions app/models/ems_refresh/refreshers/ems_refresher_mixin.rb
Original file line number Diff line number Diff line change
@@ -1,32 +1,25 @@
module EmsRefresh
module Refreshers
module EmsRefresherMixin
def format_ems_for_logging(ems)
"EMS: [#{ems.name}], id: [#{ems.id}]"
end

def refresh
_log.info "Refreshing all targets..."
preprocess_targets

@targets_by_ems_id.each do |ems_id, targets|
# Get the ems object
ems = @ems_by_ems_id[ems_id]

log_ems_target = "EMS: [#{ems.name}], id: [#{ems.id}]"
_log.info "#{log_ems_target} Refreshing targets for EMS: [#{ems.name}], id: [#{ems.id}]..."
targets.each { |t| _log.info "#{log_ems_target} #{t.class} [#{t.name}] id [#{t.id}]" }

ems_refresh_start_time = Time.now

begin
_log.debug "#{log_ems_target} Parsing #{refresher_type} inventory..."
hashes = parse_inventory(ems, targets)
_log.debug "#{log_ems_target} Parsing #{refresher_type} inventory..." \
"Completed in #{Time.now - ems_refresh_start_time} seconds"
_log.debug "#{log_ems_target} inv hashes:\n#{hashes.pretty_inspect}" if self.class::DEBUG_TRACE

if hashes.blank?
# TODO: determine if this is "success" or "failed"
_log.warn "No inventory data returned for EMS: [#{ems.name}], id: [#{ems.id}]..."
next
end
save_inventory(ems, targets, hashes)
_log.info "Refreshing all targets..."
log_ems_target = format_ems_for_logging(ems)
_log.info "#{log_ems_target} Refreshing targets for EMS..."
targets.each { |t| _log.info "#{log_ems_target} #{t.class} [#{t.name}] id [#{t.id}]" }
_, timings = Benchmark.realtime_block(:ems_refresh) { refresh_targets_for_ems(ems, targets) }
_log.info "#{log_ems_target} Refreshing targets for EMS...Complete - Timings #{timings.inspect}"
rescue => e
raise if EmsRefresh.debug_failures

Expand All @@ -52,8 +45,81 @@ def refresh
_log.info "Refreshing all targets...Complete"
end

def save_inventory(ems, _targets, hashes)
EmsRefresh.save_ems_inventory(ems, hashes)
def preprocess_targets
@full_refresh_threshold = options[:full_refresh_threshold] || 10

# See if any should be escalated to a full refresh
@targets_by_ems_id.each do |ems_id, targets|
ems = @ems_by_ems_id[ems_id]
ems_in_list = targets.any? { |t| t.kind_of?(ExtManagementSystem) }

if ems_in_list
_log.info "Defaulting to full refresh for EMS: [#{ems.name}], id: [#{ems.id}]." if targets.length > 1
targets.clear << ems
elsif targets.length >= @full_refresh_threshold
_log.info "Escalating to full refresh for EMS: [#{ems.name}], id: [#{ems.id}]."
targets.clear << ems
end
end
end

def refresh_targets_for_ems(ems, targets)
# handle a 3-part inventory refresh process
# 1. collect inventory
# 2. parse inventory
# 3. save inventory
log_header = format_ems_for_logging(ems)

targets_with_inventory, _ = Benchmark.realtime_block(:collect_inventory_for_targets) do
collect_inventory_for_targets(ems, targets)
end

until targets_with_inventory.empty?
target, inventory = targets_with_inventory.shift

_log.info "#{log_header} Refreshing target #{target.class} [#{target.name}] id [#{target.id}]..."
hashes, _ = Benchmark.realtime_block(:parse_targeted_inventory) do
parse_targeted_inventory(ems, target, inventory)
end
inventory = nil # clear to help GC

Benchmark.realtime_block(:save_inventory) { save_inventory(ems, target, hashes) }
_log.info "#{log_header} Refreshing target #{target.class} [#{target.name}] id [#{target.id}]...Complete"
end
end

def collect_inventory_for_targets(ems, targets)
# TODO: implement this method in all refreshers and remove from here
# legacy refreshers collect inventory during the parse phase so the
# inventory component of the return value is empty
# TODO: Update the docs/comment here to show the *real* bell-shaped
# targeted inventory
#
# override this method and return an array of:
# [[target1, inventory_for_target1], [target2, inventory_for_target2]]

collect_legacy_inventory_for_targets(ems)
end

def collect_legacy_inventory_for_targets(ems)
# This matches what targeted refreshers would expect for inventory
# collection. An associative array mapping the target to its inventory.
# In legacy cases the target is just the Manager, and the inventory is
# collected during the parse_legacy_inventory phase.
[[ems, nil]]
end

def parse_targeted_inventory(ems, target, inventory)
# legacy refreshers collect inventory during the parse phase
# new refreshers should override this method to parse inventory
# TODO: remove this call after all refreshers support retrieving
# inventory separate from parsing
hashes, _ = Benchmark.realtime_block(:parse_legacy_inventory) { parse_legacy_inventory(ems) }
hashes
end

def save_inventory(ems, target, hashes)
EmsRefresh.save_ems_inventory(ems, hashes, target)
end

def post_refresh_ems_cleanup(_ems, _targets)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,9 @@ module AnsibleTower
class ConfigurationManager::Refresher < ManageIQ::Providers::BaseManager::Refresher
include ::EmsRefresh::Refreshers::EmsRefresherMixin

def parse_inventory(configuration_manager, _targets)
def parse_legacy_inventory(configuration_manager)
configuration_manager.with_provider_connection do |connection|
# TODO clean up with @ems_data
configuration_manager.api_version = connection.version
configuration_manager.save
end
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ module ManageIQ::Providers::Azure
class CloudManager::Refresher < ManageIQ::Providers::BaseManager::Refresher
include ::EmsRefresh::Refreshers::EmsRefresherMixin

def parse_inventory(ems, _targets)
def parse_legacy_inventory(ems)
ManageIQ::Providers::Azure::CloudManager::RefreshParser.ems_inv_to_hashes(ems, refresher_options)
end

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,16 +3,16 @@ module Foreman
class ConfigurationManager::Refresher < ManageIQ::Providers::BaseManager::Refresher
include ::EmsRefresh::Refreshers::EmsRefresherMixin

def parse_inventory(manager, targets)
def parse_legacy_inventory(manager)
manager.with_provider_connection do |connection|
raw_ems_data = connection.inventory.refresh_configuration(targets)
raw_ems_data = connection.inventory.refresh_configuration
fetch_provisioning_manager_data(raw_ems_data, manager.provider.provisioning_manager)
ConfigurationManager::RefreshParser.configuration_inv_to_hashes(raw_ems_data)
end
end

def save_inventory(manager, targets, hashes)
EmsRefresh.save_configuration_manager_inventory(manager, hashes, targets[0])
def save_inventory(manager, target, hashes)
EmsRefresh.save_configuration_manager_inventory(manager, hashes, target)
EmsRefresh.queue_refresh(manager.provider.provisioning_manager) if hashes[:needs_provisioning_refresh]
end

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,15 +3,15 @@ module Foreman
class ProvisioningManager::Refresher < ManageIQ::Providers::BaseManager::Refresher
include ::EmsRefresh::Refreshers::EmsRefresherMixin

def parse_inventory(manager, targets)
def parse_legacy_inventory(manager)
manager.with_provider_connection do |connection|
raw_ems_data = connection.inventory.refresh_provisioning(targets)
raw_ems_data = connection.inventory.refresh_provisioning
ProvisioningManager::RefreshParser.provisioning_inv_to_hashes(raw_ems_data)
end
end

def save_inventory(manager, targets, hashes)
EmsRefresh.save_provisioning_manager_inventory(manager, hashes, targets[0])
def save_inventory(manager, target, hashes)
EmsRefresh.save_provisioning_manager_inventory(manager, hashes, target)
EmsRefresh.queue_refresh(manager.provider.configuration_manager)
end
end
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ module ManageIQ::Providers::Google
class CloudManager::Refresher < ManageIQ::Providers::BaseManager::Refresher
include ::EmsRefresh::Refreshers::EmsRefresherMixin

def parse_inventory(ems, _targets)
def parse_legacy_inventory(ems)
ManageIQ::Providers::Google::CloudManager::RefreshParser.ems_inv_to_hashes(ems, refresher_options)
end

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ module ManageIQ::Providers::Hawkular
class MiddlewareManager::Refresher < ManageIQ::Providers::BaseManager::Refresher
include ::EmsRefresh::Refreshers::EmsRefresherMixin

def parse_inventory(ems, _targets = nil)
def parse_legacy_inventory(ems)
::ManageIQ::Providers::Hawkular::MiddlewareManager::RefreshParser.ems_inv_to_hashes(ems)
end
end
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ class ContainerManager::Refresher < ManageIQ::Providers::BaseManager::Refresher
include ::EmsRefresh::Refreshers::EmsRefresherMixin
include ManageIQ::Providers::Kubernetes::ContainerManager::RefresherMixin

def parse_inventory(ems, _targets = nil)
def parse_legacy_inventory(ems)
entities = ems.with_provider_connection { |client| fetch_entities(client, KUBERNETES_ENTITIES) }
EmsRefresh.log_inv_debug_trace(entities, "inv_hash:")
ManageIQ::Providers::Kubernetes::ContainerManager::RefreshParser.ems_inv_to_hashes(entities)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ module ManageIQ::Providers::Microsoft
class InfraManager::Refresher < EmsRefresh::Refreshers::BaseRefresher
include EmsRefresh::Refreshers::EmsRefresherMixin

def parse_inventory(ems, _targets)
def parse_legacy_inventory(ems)
ManageIQ::Providers::Microsoft::InfraManager::RefreshParser.ems_inv_to_hashes(ems, refresher_options)
end

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ class ContainerManager::Refresher < ManageIQ::Providers::BaseManager::Refresher
{:name => 'build_configs'}, {:name => 'builds'}
]

def parse_inventory(ems, _targets = nil)
def parse_legacy_inventory(ems)
kube_entities = ems.with_provider_connection(:service => KUBERNETES_EMS_TYPE) do |kubeclient|
fetch_entities(kubeclient, KUBERNETES_ENTITIES)
end
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,14 @@ module ManageIQ::Providers
class Openstack::CloudManager::Refresher < ManageIQ::Providers::BaseManager::Refresher
include ::EmsRefresh::Refreshers::EmsRefresherMixin

def parse_inventory(ems, _targets)
def parse_legacy_inventory(ems)
ManageIQ::Providers::Openstack::CloudManager::RefreshParser.ems_inv_to_hashes(ems, refresher_options)
end

# TODO(lsmola) NetworkManager, remove this once we have a full representation of the NetworkManager.
# NetworkManager should refresh base on its own conditions
def save_inventory(ems, _targets, hashes)
EmsRefresh.save_ems_inventory(ems, hashes)
def save_inventory(ems, target, hashes)
super
EmsRefresh.queue_refresh(ems.network_manager)
end

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,14 @@ module Providers
class Openstack::InfraManager::Refresher < ManageIQ::Providers::BaseManager::Refresher
include ::EmsRefresh::Refreshers::EmsRefresherMixin

def parse_inventory(ems, _targets)
def parse_legacy_inventory(ems)
ManageIQ::Providers::Openstack::InfraManager::RefreshParser.ems_inv_to_hashes(ems, refresher_options)
end

# TODO(lsmola) NetworkManager, remove this once we have a full representation of the NetworkManager.
# NetworkManager should refresh base on it;s own conditions
def save_inventory(ems, _targets, hashes)
EmsRefresh.save_ems_inventory(ems, hashes)
def save_inventory(ems, target, hashes)
super
EmsRefresh.queue_refresh(ems.network_manager)
end

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ module ManageIQ::Providers
class Openstack::NetworkManager::Refresher < ManageIQ::Providers::BaseManager::Refresher
include ::EmsRefresh::Refreshers::EmsRefresherMixin

def parse_inventory(ems, _targets)
def parse_legacy_inventory(ems)
ManageIQ::Providers::Openstack::NetworkManager::RefreshParser.ems_inv_to_hashes(ems, refresher_options)
end

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,23 +2,20 @@ class ManageIQ::Providers::Redhat::InfraManager
class Refresher < ManageIQ::Providers::BaseManager::Refresher
include ::EmsRefresh::Refreshers::EmsRefresherMixin

def parse_inventory(ems, _targets)
def parse_legacy_inventory(ems)
rhevm = ems.rhevm_inventory
raise "Invalid RHEV server ip address." if rhevm.api.nil?

raw_ems_data = rhevm.refresh
return [] if raw_ems_data.blank?

#TODO cleanup with @ems_data
ems.api_version = rhevm.service.version_string
ems.save

RefreshParser.ems_inv_to_hashes(raw_ems_data)
end

def save_inventory(ems, targets, hashes)
EmsRefresh.save_ems_inventory(ems, hashes, targets[0])
end

def post_process_refresh_classes
[::VmOrTemplate, ::Host]
end
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
class ManageIQ::Providers::Vmware::InfraManager
module RefreshParser::Filter
def filter_vc_data(target)
log_header = "EMS: [#{@ems.name}], id: [#{@ems.id}]"
def filter_vc_data(ems, target)
log_header = "EMS: [#{ems.name}], id: [#{ems.id}]"

# Find the target in the data
_log.info "#{log_header} Filtering inventory for #{target.class} [#{target.name}] id: [#{target.id}]..."
Expand Down
Loading

0 comments on commit 4050485

Please sign in to comment.