Need help on deploy. #71

nnps255 · 2020-12-13T23:45:44Z

nnps255
Dec 13, 2020

Hi,

Thanks Xan for your work. Let me buy you coffees . I have tried to create the k3s cluster however seems failing since yesterday. Need ur advise the playbook. Im sorry i still quite new to ansible.

I have deployed 2 KVM(bionic) for controlplane and 4 kvm(bionic) for worker with 1 loadbalance (npm-stream to both control node k3s-master1 and k3s-master2) and 1 LXC running docker-compose with postgress database. The ansible-control machine where i play my playbook seems done but have error.

RUNNING HANDLER [kvm-k3s-deploy : restart k3s] ***************************************************************************************************************************************************************
fatal: [k3s-master1]: FAILED! => {"changed": false, "msg": "Unable to start service k3s: Job for k3s.service failed because the control process exited with error code.\nSee "systemctl status k3s.service" and "journalctl -xe" for details.\n"}

When I log in to k3s-master1 (one of my controlplane)

user@k3s-master1:$ sudo systemctl restart k3s.service
Job for k3s.service failed because the control process exited with error code.
See "systemctl status k3s.service" and "journalctl -xe" for details.
user@k3s-master1:$ systemctl status k3s.service
● k3s.service - Lightweight Kubernetes
Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Sun 2020-12-13 15:36:05 UTC; 1s ago
Docs: https://k3s.io
Process: 8573 ExecStart=/usr/local/bin/k3s server --config /etc/rancher/k3s/config.yaml (code=exited, status=1/FAILURE)
Process: 8572 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
Process: 8571 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
Main PID: 8573 (code=exited, status=1/FAILURE)
__

here is the detials of my /etc/rancher/k3s/config.yaml

cluster-cidr: 192.168.0.16/26
datastore-endpoint: postgres://k3sdb:!k3sP@ssw0rd%@datastore1.local:5432/postgres?sslmode=disable
docker: true
k3s-master1:
listen-port: 6443
k3s-master2:
listen-port: 6443
k3s-worker1: null
k3s-worker2: null
k3s-worker3: null
k3s-worker4: null

My-playbook.yml

hosts: k3s*
become: yes
become_user: root
vars:
k3s_control_node_address: npm.local # Typically a load balancer.
k3s_server:
datastore-endpoint: "postgres://k3sdb:!k3sP@ssw0rd%@datastore1.local:5432/postgres?sslmode=disable"
docker: true
cluster-cidr: 192.168.0.16/26
k3s-master1:
listen-port: 6443
k3s-master2:
listen-port: 6443
k3s_agent:
k3s-worker1:
k3s-worker2:
k3s-worker3:
k3s-worker4:
pre_tasks:
- name: Set each node to be a control node
  set_fact:
  k3s_control_node: true
  when: inventory_hostname in ['k3s-master1', 'k3s-master2']
  roles:
- role: kvm-k3s-deploy

My-Inventory.yml

k3s_cluster:
hosts:
k3s-master1:
ansible_user: nnps
ansible_host: 192.168.0.10
ansible_python_interpreter: /usr/bin/python3
k3s-master2:
ansible_user: nnps
ansible_host: 192.168.0.20
ansible_python_interpreter: /usr/bin/python3

k3s_workers:
hosts:
k3s-worker1:
ansible_user: nnps
ansible_host: 192.168.0.41
ansible_python_interpreter: /usr/bin/python3
k3s-worker2:
ansible_user: nnps
ansible_host: 192.168.0.42
ansible_python_interpreter: /usr/bin/python3
k3s-worker3:
ansible_user: nnps
ansible_host: 192.168.0.43
ansible_python_interpreter: /usr/bin/python3
k3s-worker4:
ansible_user: nnps
ansible_host: 192.168.0.44
ansible_python_interpreter: /usr/bin/python3

I belive that my variable is for playbook is wrong. Please help me with the inventory that I have. I have plan to have 2 control plane (taint-mode) with 4 worker node.

Thanks in advance.

Answered by xanmanning

Dec 16, 2020

Hi @nnps255

Sorry for the delayed response, I've been a touch busy/unwell.

Do you want to give this a try? I've not tested it but I've re-worked your config in a way that I believe should work.

`inventory.yml`

I've put all the config in here as it is convenient for sharing in GitHub discussions. You can start to separate config out of inventory if you need, either via putting it in playbooks or making use of group_vars/ or host_vars/ - see https://docs.ansible.com/ansible/latest/user_guide/playbooks_variables.html#define-variables-in-inventory for more details.

---

# This is control plane specific ("master") config
k3s_cluster:
  vars:
    k3s_control_node: true
    k3s_server:
      datas…

View full answer

xanmanning · 2020-12-16T09:04:27Z

xanmanning
Dec 16, 2020
Maintainer

Hi @nnps255

Sorry for the delayed response, I've been a touch busy/unwell.

Do you want to give this a try? I've not tested it but I've re-worked your config in a way that I believe should work.

`inventory.yml`

I've put all the config in here as it is convenient for sharing in GitHub discussions. You can start to separate config out of inventory if you need, either via putting it in playbooks or making use of group_vars/ or host_vars/ - see https://docs.ansible.com/ansible/latest/user_guide/playbooks_variables.html#define-variables-in-inventory for more details.

---

# This is control plane specific ("master") config
k3s_cluster:
  vars:
    k3s_control_node: true
    k3s_server:
      datastore-endpoint: "postgres://k3sdb:!k3sP@ssw0rd%@datastore1.local:5432/postgres?sslmode=disable"
      cluster-cidr: 192.168.0.16/26
      listen-port: 6443
      node-taint:
        - "k3s-controlplane=true:NoExecute"
  hosts:
    k3s-master1:
      ansible_user: nnps
      ansible_host: 192.168.0.10
      ansible_python_interpreter: /usr/bin/python3
    k3s-master2:
      ansible_user: nnps
      ansible_host: 192.168.0.20
      ansible_python_interpreter: /usr/bin/python3

# This is worker specific config
k3s_workers:
  vars:
    k3s_agent:
      docker: true
  hosts:
    k3s-worker1:
      ansible_user: nnps
      ansible_host: 192.168.0.41
      ansible_python_interpreter: /usr/bin/python3
    k3s-worker2:
      ansible_user: nnps
      ansible_host: 192.168.0.42
      ansible_python_interpreter: /usr/bin/python3
    k3s-worker3:
      ansible_user: nnps
      ansible_host: 192.168.0.43
      ansible_python_interpreter: /usr/bin/python3
    k3s-worker4:
      ansible_user: nnps
      ansible_host: 192.168.0.44
      ansible_python_interpreter: /usr/bin/python3

# This is global config, you could stick this in the playbook but this keeps it close to where nodes are defined.
k3s_hosts:
  vars:
    k3s_control_node_address: npm.local
  hosts:
    k3s-master1:
    k3s-master2:
    k3s-worker1:
    k3s-worker2:
    k3s-worker3:
    k3s-worker4:

`playbook.yml`

I've simplified this, you can therefore have multiple inventory.yml files configured to suit your needs and run the same playbook.yml against it.

---

  - hosts: k3s_hosts
    become: yes
    become_user: root
    roles:
      role: kvm-k3s-deploy

2 replies

funnym0nk3y Jan 30, 2021

What is the difference between k3s_cluster, k3s_workers and k3s_hosts? Are those just arbitraty names for the groups or do they have a meaning?

xanmanning Jan 31, 2021
Maintainer

Hi @funnym0nk3y , they are arbitrary group names that @nnps255 defined.

A slightly easier to read and simplified inventory.yml and playbook.yml might be as per the bellow, where I have used the prefix hostgroup_:

`inventory.yml`

---

# This is control plane specific ("master") config
hostgroup_k3s_control_plane:
  vars:
    k3s_control_node: true
    k3s_server:
      datastore-endpoint: "postgres://k3sdb:!k3sP@ssw0rd%@datastore1.local:5432/postgres?sslmode=disable"
      cluster-cidr: 192.168.0.16/26
      https-listen-port: 6443
      node-taint:
        - "k3s-controlplane=true:NoExecute"
  hosts:
    k3s-master1:
      ansible_user: nnps
      ansible_host: 192.168.0.10
      ansible_python_interpreter: /usr/bin/python3
    k3s-master2:
      ansible_user: nnps
      ansible_host: 192.168.0.20
      ansible_python_interpreter: /usr/bin/python3

# This is worker specific config
hostgroup_k3s_workers:
  vars:
    k3s_agent:
      docker: true
  hosts:
    k3s-worker1:
      ansible_user: nnps
      ansible_host: 192.168.0.41
      ansible_python_interpreter: /usr/bin/python3
    k3s-worker2:
      ansible_user: nnps
      ansible_host: 192.168.0.42
      ansible_python_interpreter: /usr/bin/python3
    k3s-worker3:
      ansible_user: nnps
      ansible_host: 192.168.0.43
      ansible_python_interpreter: /usr/bin/python3
    k3s-worker4:
      ansible_user: nnps
      ansible_host: 192.168.0.44
      ansible_python_interpreter: /usr/bin/python3

# This is global config, you could stick this in the playbook but this keeps it close to where nodes are defined.
hostgroup_k3s_hosts:
  vars:
    k3s_control_node_address: npm.local
  children:
    hostgroup_k3s_control_plane:
    hostgroup_k3s_workers:

`playbook.yml`

---

  - hosts: hostgroup_k3s_hosts
    become: yes
    become_user: root
    roles:
      role: kvm-k3s-deploy

nnps255 · 2020-12-19T13:17:57Z

nnps255
Dec 19, 2020
Author

Hey Thanks Xan,

I definitely will try this.. I am busy with glusterfs.. just done some deployment. I will let you know the outcome. Btw thank you very very much for your reply.. Feel exited to try it soon!

9 replies

praveenmak Feb 19, 2021

@xanmanning , let me explain.

I just cloned your repo like few days back. No customizations. k3s_state is set to default

k3s_state: installed

This is how my inventory.yml looks like.

# This is control plane specific ("master") config
k3s_masters:
  vars:
    k3s_etcd_datastore: true
    k3s_server:
      cluster-cidr: 10.220.112.144/28
      listen-port: 6443
      node-taint:
        - "k3s-controlplane=true:NoExecute"
  hosts:
    kesp-master-01:
      ansible_user: centos
      ansible_host: 10.220.112.158
      ansible_python_interpreter: /usr/bin/python3
      k3s_control_node: true

# This is worker specific config
k3s_workers:
  vars:
    k3s_agent:
      docker: true
  hosts:
    kesp-worker-01:
      ansible_user: centos
      ansible_host: 10.220.112.157
      ansible_python_interpreter: /usr/bin/python3
      k3s_control_node: true
    kesp-worker-02:
      ansible_user: centos
      ansible_host: 10.220.112.149
      ansible_python_interpreter: /usr/bin/python3
      k3s_control_node: true


# This is global config, you could stick this in the playbook but this keeps it close to where nodes are defined.
k3s_hosts:
  vars:
    k3s_control_node_address: 10.220.112.158
  children:
    k3s_masters:
    k3s_workers:

praveenmak Feb 19, 2021

@xanmanning

I was able to fix that issue.

I commented the following line in tasks/state-installed.yml and that fixed everything. I had to do slight modifications in inventory.yml.

import_tasks: teardown/drain-and-remove-nodes.yml

Again, kudos to you for building this playbook. Beer is a good idea 🥇

xanmanning Feb 20, 2021
Maintainer

I'm still not sure how your config works, couple of things. K3S v1.20, you are going to get an error with this config because you've not set: k3s_use_unsupported_config: true

failed: [kesp-worker-01] (item={'setting': 'docker', 'correction': 'docker: false', 'when': '1.20.0'}) => {
    "ansible_loop_var": "item",
    "assertion": "(item.setting not in k3s_runtime_config) or (not k3s_runtime_config[item.setting])",
    "changed": false,
    "evaluated_to": false,
    "item": {
        "correction": "docker: false",
        "setting": "docker",
        "when": "1.20.0"
    },
    "msg": "docker found in server/agent config. Please set docker: false to use this option. "
}
failed: [kesp-worker-02] (item={'setting': 'docker', 'correction': 'docker: false', 'when': '1.20.0'}) => {
    "ansible_loop_var": "item",
    "assertion": "(item.setting not in k3s_runtime_config) or (not k3s_runtime_config[item.setting])",
    "changed": false,
    "evaluated_to": false,
    "item": {
        "correction": "docker: false",
        "setting": "docker",
        "when": "1.20.0"
    },
    "msg": "docker found in server/agent config. Please set docker: false to use this option. "
}

Then... setting k3s_use_unsupported_config: true, you fall into this issue: Check the conditions when multiple controllers are defined

ok: [kesp-master-01] => {
    "changed": false,
    "msg": "Control plane configuration is valid."
}
fatal: [kesp-worker-01]: FAILED! => {
    "assertion": "((\"datastore-endpoint\" in k3s_runtime_config and k3s_runtime_config['datastore-endpoint']) or (k3s_etcd_datastore is defined and k3s_etcd_datastore))",
    "changed": false,
    "evaluated_to": false,
    "msg": "Control plane configuration is invalid. Please see notes about k3s_control_node and HA in README.md."
}
fatal: [kesp-worker-02]: FAILED! => {
    "assertion": "((\"datastore-endpoint\" in k3s_runtime_config and k3s_runtime_config['datastore-endpoint']) or (k3s_etcd_datastore is defined and k3s_etcd_datastore))",
    "changed": false,
    "evaluated_to": false,
    "msg": "Control plane configuration is invalid. Please see notes about k3s_control_node and HA in README.md."
}

This is because you've defined k3s_workers as a separate group of "control nodes" (k3s_control_node: true) but not specified the k3s_etcd_datastore: true for these. There's 2 nodes in this group and without k3s_etcd_datastore it is assumed you should be using an external datastore such as PostgreSQL or MySQL.

On top of this, I copied from the original post by @nnps255 which had a typo (It's https-listen-port, not listen-port).

I kind of got it working with the following config, but had a few errors restarting the k3s service on worker nodes (I don't think they joined).

discussion-71.tar.gz

praveenmak Feb 21, 2021

That is correct, as I had mentioned in my earlier message. I had to do slight modifications to inventory.yml

Here is what is working for me. (I noticed you have set k3s_become_for_all , for me centos user is baked in as super user by default, I don't need this)


k3s_masters:
  vars:
    k3s_etcd_datastore: true
    k3s_server:
      https-listen-port: 6443
      node-taint:
        - "k3s-controlplane=true:NoExecute"
  hosts:
    kesp-master-01:
      ansible_user: centos
      ansible_host: 10.220.112.158
      ansible_python_interpreter: /usr/bin/python3
      k3s_control_node: true

k3s_workers:
  vars:
    k3s_etcd_datastore: true
    k3s_agent:
      docker: false
  hosts:
    kesp-worker-01:
      ansible_user: centos
      ansible_host: 10.220.112.157
      ansible_python_interpreter: /usr/bin/python3
      k3s_control_node: true
    kesp-worker-02:
      ansible_user: centos
      ansible_host: 10.220.112.149
      ansible_python_interpreter: /usr/bin/python3
      k3s_control_node: true

k3s_hosts:
  vars:
    k3s_control_node_address: 10.220.112.158
  hosts:
    kesp-master-01:
    kesp-worker-01:
    kesp-worker-02:

praveenmak Feb 21, 2021

By the way I ran into another issue.

When I go to control node address (10.220.112.158) and use any kubectl command kubectl get nodes/ns etc , it takes 15+ seconds to get a response.

There is something wrong with the cluster. I have tried Rancher ansible (non-HA) and no issues with that.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Need help on deploy. #71

{{title}}

Replies: 2 comments 11 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Need help on deploy. #71

nnps255 Dec 13, 2020

inventory.yml

Replies: 2 comments · 11 replies

xanmanning Dec 16, 2020 Maintainer

inventory.yml

playbook.yml

funnym0nk3y Jan 30, 2021

xanmanning Jan 31, 2021 Maintainer

inventory.yml

playbook.yml

nnps255 Dec 19, 2020 Author

praveenmak Feb 19, 2021

This is how my inventory.yml looks like.

praveenmak Feb 19, 2021

xanmanning Feb 20, 2021 Maintainer

praveenmak Feb 21, 2021

praveenmak Feb 21, 2021

nnps255
Dec 13, 2020

`inventory.yml`

Replies: 2 comments 11 replies

xanmanning
Dec 16, 2020
Maintainer

`inventory.yml`

`playbook.yml`

xanmanning Jan 31, 2021
Maintainer

`inventory.yml`

`playbook.yml`

nnps255
Dec 19, 2020
Author

xanmanning Feb 20, 2021
Maintainer