Live Migration role has failed tasks on Ubuntu 16.04 with CentOS hosts included

[mmccarre]

SUMMARY

ssh keys do not get stored properly on Ubuntu 16.04 nodes when CentOS hypervisors are also on-boarded

Low priority issue since it's uncommon to see mixed OS usage for hypervisors.

ISSUE TYPE

• Bug Report

COMPONENT NAME

secure-live-migration role

ANSIBLE VERSION

ansible 2.8.4
  config file = /opt/pf9-express/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /bin/ansible
  python version = 2.7.5 (default, Jun 20 2019, 20:27:34) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]

CONFIGURATION

ANSIBLE_NOCOWS(/opt/pf9-express/ansible.cfg) = True
ANSIBLE_PIPELINING(/opt/pf9-express/ansible.cfg) = True
COMMAND_WARNINGS(/opt/pf9-express/ansible.cfg) = False
DEFAULT_CALLBACK_WHITELIST(/opt/pf9-express/ansible.cfg) = [u'profile_tasks']
DEFAULT_HOST_LIST(/opt/pf9-express/ansible.cfg) = [u'/opt/pf9-express/inventory/hosts']
DEPRECATION_WARNINGS(/opt/pf9-express/ansible.cfg) = False
HOST_KEY_CHECKING(/opt/pf9-express/ansible.cfg) = False
RETRY_FILES_ENABLED(/opt/pf9-express/ansible.cfg) = False

OS / ENVIRONMENT

Ubuntu 16.04 hypervisors / CentOS 7.4 Express control host

STEPS TO REPRODUCE

Enroll two Ubuntu 16.04 hypervisors. Example below includes one CentOS 7.6 host (rt-hv-05)

ACTUAL RESULTS

TASK [secure-live-migration : Install sysfsutils] **************************************************************************************************************
Wednesday 28 August 2019  18:54:21 +0000 (0:00:00.354)       0:08:36.900 ****** 
ok: [rt-hv05]
ok: [rt-hv03]
ok: [rt-hv04]

TASK [secure-live-migration : Tell libvirt to listen on startup - Red Hat] *************************************************************************************
Wednesday 28 August 2019  18:54:23 +0000 (0:00:01.781)       0:08:38.682 ****** 
skipping: [rt-hv03]
skipping: [rt-hv04]
changed: [rt-hv05]

TASK [secure-live-migration : Tell libvirt to listen on startup - Ubuntu] **************************************************************************************
Wednesday 28 August 2019  18:54:24 +0000 (0:00:00.728)       0:08:39.410 ****** 
skipping: [rt-hv05]
changed: [rt-hv03]
changed: [rt-hv04]

TASK [secure-live-migration : Create /etc/pf9/certs/libvirt] ***************************************************************************************************
Wednesday 28 August 2019  18:54:25 +0000 (0:00:00.787)       0:08:40.198 ****** 
changed: [rt-hv03]
changed: [rt-hv04]
changed: [rt-hv05]

TASK [secure-live-migration : Link PF9 certs into /etc/pf9/certs/libvirt] **************************************************************************************
Wednesday 28 August 2019  18:54:25 +0000 (0:00:00.799)       0:08:40.998 ****** 
changed: [rt-hv03] => (item={u'src': u'ca/cert.pem', u'name': u'cacert.pem'})
changed: [rt-hv04] => (item={u'src': u'ca/cert.pem', u'name': u'cacert.pem'})
changed: [rt-hv05] => (item={u'src': u'ca/cert.pem', u'name': u'cacert.pem'})
changed: [rt-hv03] => (item={u'src': u'hostagent/cert.pem', u'name': u'clientcert.pem'})
changed: [rt-hv05] => (item={u'src': u'hostagent/cert.pem', u'name': u'clientcert.pem'})
changed: [rt-hv04] => (item={u'src': u'hostagent/cert.pem', u'name': u'clientcert.pem'})
changed: [rt-hv03] => (item={u'src': u'hostagent/key.pem', u'name': u'clientkey.pem'})
changed: [rt-hv05] => (item={u'src': u'hostagent/key.pem', u'name': u'clientkey.pem'})
changed: [rt-hv04] => (item={u'src': u'hostagent/key.pem', u'name': u'clientkey.pem'})

TASK [secure-live-migration : Set libvirtd configuration] ******************************************************************************************************
Wednesday 28 August 2019  18:54:27 +0000 (0:00:01.729)       0:08:42.727 ****** 
changed: [rt-hv03] => (item={'value': u'"none"', 'key': u'auth_tls'})
changed: [rt-hv05] => (item={'value': u'"none"', 'key': u'auth_tls'})
changed: [rt-hv04] => (item={'value': u'"none"', 'key': u'auth_tls'})
changed: [rt-hv03] => (item={'value': 1, 'key': u'listen_tls'})
changed: [rt-hv05] => (item={'value': 1, 'key': u'listen_tls'})
changed: [rt-hv04] => (item={'value': 1, 'key': u'listen_tls'})
changed: [rt-hv03] => (item={'value': u'"sasl"', 'key': u'auth_tcp'})
changed: [rt-hv05] => (item={'value': u'"sasl"', 'key': u'auth_tcp'})
changed: [rt-hv04] => (item={'value': u'"sasl"', 'key': u'auth_tcp'})
changed: [rt-hv03] => (item={'value': 0, 'key': u'listen_tcp'})
changed: [rt-hv05] => (item={'value': 0, 'key': u'listen_tcp'})
changed: [rt-hv04] => (item={'value': 0, 'key': u'listen_tcp'})
changed: [rt-hv03] => (item={'value': u'"/etc/pf9/certs/libvirt/cacert.pem"', 'key': u'ca_file'})
changed: [rt-hv05] => (item={'value': u'"/etc/pf9/certs/libvirt/cacert.pem"', 'key': u'ca_file'})
changed: [rt-hv04] => (item={'value': u'"/etc/pf9/certs/libvirt/cacert.pem"', 'key': u'ca_file'})
changed: [rt-hv05] => (item={'value': u'"/etc/pf9/certs/libvirt/clientkey.pem"', 'key': u'key_file'})
changed: [rt-hv03] => (item={'value': u'"/etc/pf9/certs/libvirt/clientkey.pem"', 'key': u'key_file'})
changed: [rt-hv04] => (item={'value': u'"/etc/pf9/certs/libvirt/clientkey.pem"', 'key': u'key_file'})
changed: [rt-hv05] => (item={'value': u'"/etc/pf9/certs/libvirt/clientcert.pem"', 'key': u'cert_file'})
changed: [rt-hv03] => (item={'value': u'"/etc/pf9/certs/libvirt/clientcert.pem"', 'key': u'cert_file'})
changed: [rt-hv04] => (item={'value': u'"/etc/pf9/certs/libvirt/clientcert.pem"', 'key': u'cert_file'})

TASK [secure-live-migration : Set libvirt live migration settings in Nova.conf] ********************************************************************************
Wednesday 28 August 2019  18:54:30 +0000 (0:00:03.228)       0:08:45.956 ****** 
changed: [rt-hv05] => (item={u'option': u'live_migration_uri', u'value': u'qemu+tls://%s/system?no_verify=1&pkipath=/etc/pf9/certs/libvirt'})
changed: [rt-hv04] => (item={u'option': u'live_migration_uri', u'value': u'qemu+tls://%s/system?no_verify=1&pkipath=/etc/pf9/certs/libvirt'})
changed: [rt-hv03] => (item={u'option': u'live_migration_uri', u'value': u'qemu+tls://%s/system?no_verify=1&pkipath=/etc/pf9/certs/libvirt'})
changed: [rt-hv05] => (item={u'option': u'live_migration_flag', u'value': u'VIR_MIGRATE_UNDEFINE_SOURCE, VIR_MIGRATE_PEER2PEER, VIR_MIGRATE_LIVE, VIR_MIGRATE_TUNNELLED'})
changed: [rt-hv03] => (item={u'option': u'live_migration_flag', u'value': u'VIR_MIGRATE_UNDEFINE_SOURCE, VIR_MIGRATE_PEER2PEER, VIR_MIGRATE_LIVE, VIR_MIGRATE_TUNNELLED'})
changed: [rt-hv04] => (item={u'option': u'live_migration_flag', u'value': u'VIR_MIGRATE_UNDEFINE_SOURCE, VIR_MIGRATE_PEER2PEER, VIR_MIGRATE_LIVE, VIR_MIGRATE_TUNNELLED'})

TASK [secure-live-migration : Configure libvirtd to start on boot] *********************************************************************************************
Wednesday 28 August 2019  18:54:32 +0000 (0:00:01.786)       0:08:47.742 ****** 
ok: [rt-hv03]
ok: [rt-hv05]
ok: [rt-hv04]

TASK [secure-live-migration : Enable & start SSH] **************************************************************************************************************
Wednesday 28 August 2019  18:54:33 +0000 (0:00:01.060)       0:08:48.803 ****** 
ok: [rt-hv03]
ok: [rt-hv05]
ok: [rt-hv04]

TASK [secure-live-migration : Configure pf9 user on host] ******************************************************************************************************
Wednesday 28 August 2019  18:54:34 +0000 (0:00:01.051)       0:08:49.854 ****** 
changed: [rt-hv03]
changed: [rt-hv05]
changed: [rt-hv04]

TASK [secure-live-migration : Create ~pf9/.ssh/config] *********************************************************************************************************
Wednesday 28 August 2019  18:54:39 +0000 (0:00:04.515)       0:08:54.370 ****** 
changed: [rt-hv03]
changed: [rt-hv05]
changed: [rt-hv04]

TASK [secure-live-migration : Read pf9 user's public key] ******************************************************************************************************
Wednesday 28 August 2019  18:54:40 +0000 (0:00:01.285)       0:08:55.655 ****** 
ok: [rt-hv05]
ok: [rt-hv04]
ok: [rt-hv03]

TASK [secure-live-migration : Add authorized_key to other nodes] ***********************************************************************************************
Wednesday 28 August 2019  18:54:41 +0000 (0:00:01.429)       0:08:57.084 ****** 
skipping: [rt-hv03] => (item=rt-hv03) 
changed: [rt-hv05 -> 10.128.233.197] => (item=rt-hv03)
changed: [rt-hv03 -> 10.128.233.152] => (item=rt-hv04)
changed: [rt-hv04 -> 10.128.233.197] => (item=rt-hv03)
skipping: [rt-hv04] => (item=rt-hv04) 
failed: [rt-hv03 -> 10.128.233.242] (item=rt-hv05) => {"ansible_loop_var": "item", "changed": false, "item": "rt-hv05", "module_stderr": "/bin/sh: /usr/bin/python3: No such file or directory\n", "module_stdout": "", "msg": "The module failed to execute correctly, you probably need to set the interpreter.\nSee stdout/stderr for the exact error", "rc": 127}
failed: [rt-hv04 -> 10.128.233.242] (item=rt-hv05) => {"ansible_loop_var": "item", "changed": false, "item": "rt-hv05", "module_stderr": "/bin/sh: /usr/bin/python3: No such file or directory\n", "module_stdout": "", "msg": "The module failed to execute correctly, you probably need to set the interpreter.\nSee stdout/stderr for the exact error", "rc": 127}
changed: [rt-hv05 -> 10.128.233.152] => (item=rt-hv04)
skipping: [rt-hv05] => (item=rt-hv05) 

TASK [secure-live-migration : Add SSH keys for host(s)] ********************************************************************************************************
Wednesday 28 August 2019  18:54:44 +0000 (0:00:02.224)       0:08:59.309 ****** 
included: /opt/pf9-express/roles/secure-live-migration/tasks/add-keys-for-host.yaml for rt-hv05

TASK [secure-live-migration : Add SSH keys for rt-hv05 by FQDN] ************************************************************************************************
Wednesday 28 August 2019  18:54:44 +0000 (0:00:00.269)       0:08:59.578 ****** 
changed: [rt-hv05] => (item={u'type': u'ssh-rsa', u'name': u'ansible_ssh_host_key_rsa_public'})
changed: [rt-hv05] => (item={u'type': u'ecdsa-sha2-nistp256', u'name': u'ansible_ssh_host_key_ecdsa_public'})

TASK [secure-live-migration : Add SSH host keys for rt-hv05 by IP] *********************************************************************************************
Wednesday 28 August 2019  18:54:45 +0000 (0:00:01.347)       0:09:00.925 ****** 
changed: [rt-hv05] => (item={u'type': u'ssh-rsa', u'name': u'ansible_ssh_host_key_rsa_public'})
changed: [rt-hv05] => (item={u'type': u'ecdsa-sha2-nistp256', u'name': u'ansible_ssh_host_key_ecdsa_public'})

RUNNING HANDLER [secure-live-migration : restart libvirtd] *****************************************************************************************************
Wednesday 28 August 2019  18:54:47 +0000 (0:00:00.037)       0:09:02.397 ****** 
changed: [rt-hv05]

RUNNING HANDLER [secure-live-migration : restart pf9-ostackhost] ***********************************************************************************************
Wednesday 28 August 2019  18:54:48 +0000 (0:00:01.524)       0:09:03.921 ****** 
changed: [rt-hv05]