Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to change the NOOBAA_LOG_LEVEL on the ODF 4.9.2-9 as noobaa-endpoint doesn't restart #851

Closed
rkomandu opened this issue Feb 3, 2022 · 5 comments
Closed

Unable to change the NOOBAA_LOG_LEVEL on the ODF 4.9.2-9 as noobaa-endpoint doesn't restart #851

rkomandu opened this issue Feb 3, 2022 · 5 comments
Labels
NS-FS Request Validation

Comments

@rkomandu
Copy link

rkomandu commented Feb 3, 2022

Environment info

  • NooBaa Operator Version: VERSION
  • Platform: Kubernetes 1.14.1 | minikube 1.1.1 | OpenShift 4.1 | other: specify

oc version
Client Version: 4.9.5
Server Version: 4.9.5
Kubernetes Version: v1.22.0-rc.0+a44d0f0

ODF 4.9.2-9 downstream builds

Actual behavior

 noobaa version
INFO[0000] CLI version: 5.9.2
INFO[0000] noobaa-image: noobaa/noobaa-core:5.9.0
INFO[0000] operator-image: noobaa/noobaa-operator:5.9.2


noobaa status
INFO[0000] CLI version: 5.9.2
INFO[0000] noobaa-image: quay.io/rhceph-dev/odf4-mcg-core-rhel8@sha256:5507f2c1074bfb023415f0fef16ec42fbe6e90c540fc45f1111c8c929e477910
INFO[0000] operator-image: quay.io/rhceph-dev/odf4-mcg-rhel8-operator@sha256:b314ad9f15a10025bade5c86857a7152c438b405fdba26f64826679a5c5bff1b
INFO[0000] noobaa-db-image: quay.io/rhceph-dev/rhel8-postgresql-12@sha256:623bdaa1c6ae047db7f62d82526220fac099837afd8770ccc6acfac4c7cff100
INFO[0000] Namespace: openshift-storage
...





NAME                                               READY   STATUS    RESTARTS        AGE     IP              NODE                                  NOMINATED N
ODE   READINESS GATES
noobaa-core-0                                      1/1     Running   0               2d20h   10.254.23.179   worker2.rkomandu-ta.cp.fyre.ibm.com   <none>
      <none>
noobaa-db-pg-0                                     1/1     Running   0               47h     10.254.23.217   worker2.rkomandu-ta.cp.fyre.ibm.com   <none>
      <none>
noobaa-default-backing-store-noobaa-pod-77176233   1/1     Running   0               2d4h    10.254.18.15    worker0.rkomandu-ta.cp.fyre.ibm.com   <none>
      <none>
noobaa-endpoint-5f86849854-7mgjl                   1/1     Running   0               47h     10.254.12.8     worker1.rkomandu-ta.cp.fyre.ibm.com   <none>
      <none>
noobaa-endpoint-5f86849854-pttcg                   1/1     Running   0               2d20h   10.254.23.172   worker2.rkomandu-ta.cp.fyre.ibm.com   <none>
      <none>
noobaa-endpoint-5f86849854-zn8nx                   1/1     Running   0               2d20h   10.254.16.225   worker0.rkomandu-ta.cp.fyre.ibm.com   <none>
      <none>
noobaa-operator-54877b7dc9-zjsvl                   1/1     Running   0               2d      10.254.18.86    worker0.rkomandu-ta.cp.fyre.ibm.com   <none>
      <none>
ocs-metrics-exporter-7955bfc785-cn2zl              1/1     Running   0               2d      10.254.18.84    worker0.rkomandu-ta.cp.fyre.ibm.com   <none>
      <none>
ocs-operator-57d785c8c7-bqpfl                      1/1     Running   12 (178m ago)   2d      10.254.18.90    worker0.rkomandu-ta.cp.fyre.ibm.com   <none>
      <none>
odf-console-756c9c8bc7-4jsfl                       1/1     Running   0               2d      10.254.18.88    worker0.rkomandu-ta.cp.fyre.ibm.com   <none>
      <none>
odf-operator-controller-manager-89746b599-z64f6    2/2     Running   12 (178m ago)   2d      10.254.18.87    worker0.rkomandu-ta.cp.fyre.ibm.com   <none>
      <none>
rook-ceph-operator-74864f7c6f-rlf6c                1/1     Running   0               2d      10.254.18.82    worker0.rkomandu-ta.cp.fyre.ibm.com   <none>
      <none>

oc edit configmap/noobaa-config
apiVersion: v1
data:
  DISABLE_DEV_RANDOM_SEED: "true"
  NOOBAA_DISABLE_COMPRESSION: "false"
  NOOBAA_LOG_LEVEL: nsfs
kind: ConfigMap
metadata:
  annotations:
    noobaa.io/configmap-hash: 109221e924cc07b59e9d6ad4d40cb1438d2836cc34345266f8ee41258009cf78
  creationTimestamp: "2022-01-27T05:18:27Z"
  labels:
    app: noobaa
  name: noobaa-config
  namespace: openshift-storage
....
 resourceVersion: "39003257"
  uid: bf657532-d774-4237-b904-3f5fca45d68d



Noobaa-endpoint logs doesn't show any L1/L2/L3/L4 levels once the LOG_LEVEL is changed 

[[email protected] ~]# grep L1 /tmp/log-endpoint
[[email protected] ~]# grep L0 /tmp/log-endpoint|tail
Feb-3 8:20:14.592 [Endpoint/14]    [L0] core.endpoint.endpoint:: Sending endpoint report: { group_name: 'a589cd92-09f9-4a68-9a36-fac7f528702f', hostname: 'noobaa-endpoint-5f86849854-zn8nx' }
Feb-3 8:30:14.693 [Endpoint/14]    [L0] core.endpoint.endpoint:: Sending endpoint report: { group_name: 'a589cd92-09f9-4a68-9a36-fac7f528702f', hostname: 'noobaa-endpoint-5f86849854-zn8nx' }
Feb-3 8:40:14.808 [Endpoint/14]    [L0] core.endpoint.endpoint:: Sending endpoint report: { group_name: 'a589cd92-09f9-4a68-9a36-fac7f528702f', hostname: 'noobaa-endpoint-5f86849854-zn8nx' }
Feb-3 8:50:14.893 [Endpoint/14]    [L0] core.endpoint.endpoint:: Sending endpoint report: { group_name: 'a589cd92-09f9-4a68-9a36-fac7f

528702f', hostname: 'noobaa-endpoint-5f86849854-zn8nx' }
Feb-3 9:00:14.997 [Endpoint/14]    [L0] core.endpoint.endpoint:: Sending endpoint report: { group_name: 'a589cd92-09f9-4a68-9a36-fac7f528702f', hostname: 'noobaa-endpoint-5f86849854-zn8nx' }
Feb-3 9:10:15.099 [Endpoint/14]    [L0] core.endpoint.endpoint:: Sending endpoint report: { group_name: 'a589cd92-09f9-4a68-9a36-fac7f528702f', hostname: 'noobaa-endpoint-5f86849854-zn8nx' }
Feb-3 9:20:15.135 [Endpoint/14]    [L0] core.endpoint.endpoint:: Sending endpoint report: { group_name: 'a589cd92-09f9-4a68-9a36-fac7f528702f', hostname: 'noobaa-endpoint-5f86849854-zn8nx' }
Feb-3 9:30:15.195 [Endpoint/14]    [L0] core.endpoint.endpoint:: Sending endpoint report: { group_name: 'a589cd92-09f9-4a68-9a36-fac7f528702f', hostname: 'noobaa-endpoint-5f86849854-zn8nx' }
Feb-3 9:40:15.342 [Endpoint/14]    [L0] core.endpoint.endpoint:: Sending endpoint report: { group_name: 'a589cd92-09f9-4a68-9a36-fac7f528702f', hostname: 'noobaa-endpoint-5f86849854-zn8nx' }
Feb-3 9:50:15.396 [Endpoint/14]    [L0] core.endpoint.endpoint:: Sending endpoint report: { group_name: 'a589cd92-09f9-4a68-9a36-fac7f528702f', hostname: 'noobaa-endpoint-5f86849854-zn8nx' }

Expected behavior

Should be able to change the Log_level

Steps to reproduce

Edit the configmap of noobaa-config

oc edit configmap/noobaa-config

Tried other ways as per discussion with Liran Mauda and no luck

oc rsh noobaa-endpoint-5f86849854-zn8nx
sh-4.4# env |grep LOG_LEVEL
NOOBAA_LOG_LEVEL=default_level
sh-4.4# export NOOBAA_LOG_LEVEL=nsfs
sh-4.4# env |grep LOG_LEVEL
NOOBAA_LOG_LEVEL=nsfs
sh-4.4# uname -a
Linux noobaa-endpoint-5f86849854-zn8nx 4.18.0-305.19.1.el8_4.x86_64 #1 SMP Tue Sep 7 07:07:31 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux
sh-4.4# quit
sh: quit: command not found
sh-4.4# exit
exit
command terminated with exit code 127

More information - Screenshots / Logs / Other output

must-gather.local-noobaa-log-level.tar.gz
@rkomandu
Copy link
Author

rkomandu commented Feb 4, 2022

@liranmauda , this could be due to the noobaa/noobaa-core#6869 as the noobaa log level was able to set post the noobaa-core pod restart.

The defect 6869 has to be fixed as a priority

@liranmauda
Copy link
Contributor

@rkomandu
When the pods are not able to get a restart, the env available will not change.
We did see that the annotation of noobaa.io/configmap-hash was correct (the same in every resource).

We can close this issue.

@rkomandu
Copy link
Author

rkomandu commented Feb 7, 2022

@liranmauda , once the defect fix for 6869 is verified then we can close this issue IMO.

@rkomandu
Copy link
Author

rkomandu commented Feb 7, 2022

We should have low priority for this defect now.

@rkomandu
Copy link
Author

rkomandu commented Apr 7, 2022 •
edited
Loading

Could see endpoint restarted as expected. This is tried on ODF 4.9.5-4 d/s builds. Earlier the problem could be a combination of defect #noobaa-core/6870.

NAME                                               READY   STATUS    RESTARTS        AGE     IP             NODE                                   NOMINATED NODE   RE
ADINESS GATES
noobaa-core-0                                      1/1     Running   0               2m21s   10.254.16.34   worker0.rkomandu-513.cp.fyre.ibm.com   <none>           <n
one>
noobaa-db-pg-0                                     1/1     Running   0               22h     10.254.20.44   worker2.rkomandu-513.cp.fyre.ibm.com   <none>           <n
one>
noobaa-default-backing-store-noobaa-pod-e5452524   1/1     Running   0               100m    10.254.12.40   worker1.rkomandu-513.cp.fyre.ibm.com   <none>           <n
one>
noobaa-endpoint-6d89cbb64f-cxvdm                   1/1     Running   0               2m36s   10.254.12.58   worker1.rkomandu-513.cp.fyre.ibm.com   <none>           <n
one>
noobaa-endpoint-6d89cbb64f-kjck8                   1/1     Running   0               2m47s   10.254.16.33   worker0.rkomandu-513.cp.fyre.ibm.com   <none>           <n
one>
noobaa-endpoint-6d89cbb64f-swvgd                   1/1     Running   0               2m52s   10.254.16.32   worker0.rkomandu-513.cp.fyre.ibm.com   <none>           <n
one>

closing this for now

@rkomandu rkomandu closed this as completed Apr 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
NS-FS Request Validation
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@rkomandu @liranmauda