Error missing label of app on certain deployment #264
Comments
|
Hey, if you run krr with |
pavangudiwada
changed the title
|
Hey, would it be possible to provide more information (e.g. the app label from the problematic pod) to help us fix this? Without more information, we're going to close until we can replicate. |
|
getting similar error : Getting this error for almost all k8s resource jobs, cronjobs, daemonsets, statefulsets, rollout and deployment. The label on these resource is as simple as |
|
What is the full CLI command that you ran krr with? Are you passing a label selector? |
|
I am executing this command similar error with brew. |
|
Does this also occur on the branch |
|
And if that still does not solve the problem, please try the branch We aren't able to replicate ourselves, but with your help I hope that we can get to the bottom of this! |
|
Hi @aantn Thanks for your help ! |
|
Any chance you can still run |
|
@aantn I just had the same issue on v1.11.0. I tried to run the following command on both |
|
Thanks, on the branch The contents of that log will help us figure out what the issue is. |
|
Yep, I do (sorry I had to anonymise most of the names :-( ): What I can also tell you, which maybe is related, is that even if the JSON export is working if I select a subset of namespaces, the generated JSON is usually not valid. Somewhere on the JSON (depending on which namespace I execute) I always have this description, which breaks the JSON validity: This is not limited to any of the branch above. All of them have the same issue as far as I tested. |
|
Thank you. Did you include all the matching log lines or only some of them? I am particularly interested in the last log line before the exception. I am trying to figure out which listing of pods had an invalid app.kubernetes.io/component value and why. From your original log: The mystery is what was the label value that broke things and how it is possible. Regarding the JSON export, can you please open a separate ticket? |
Actually I missed only one, which is the last one before the exception: And then comes the exception posted above.
Yep, I will |
|
Thank you, we are very close to fixing this. I've narrowed it down to the problematic code, but I am still unable to reproduce myself. What is the kind of the Kubernetes workload (e.g. Deployment, StatefulSet) and what are the contents of spec.matchLabels? |
|
Sorry, I mean what are the contents of |
|
Actually it's a Rollout (from Argo Rollouts) but it's very close to a Deployment and I don't think it makes a difference. The content of |
|
Thanks, that was actually very important information! The kubernetes python client renames matchLabels to I have created fix here - #308 |
|
Yes I can confirm that it is working with Rollouts 👍 |
|
Wonderful, thanks for the confirmation. I've merged the changes into the |
Describe the bug
i have deployed KRR in my kubernetes cluster
while running a simple test getting error which says the label of app on certain deployment is missing
not able to to find which deployment is that
Attaching the error message
also i am receiving this error
ERROR An unexpected error occurred runner.py:332
Traceback (most recent call last):
File "/Users/namanjain/Documents/krr/robusta_krr/core/runner.py", line 325, in run
result = await self._collect_result()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/namanjain/Documents/krr/robusta_krr/core/runner.py", line 279, in _collect_result
scans = await asyncio.gather([self._gather_object_allocations(k8s_object) for k8s_object in workloads])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/namanjain/Documents/krr/robusta_krr/core/runner.py", line 243, in _gather_object_allocations
recommendation = await self._calculate_object_recommendations(k8s_object)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/namanjain/Documents/krr/robusta_krr/core/runner.py", line 177, in calculate_object_recommendations
object.pods = await self.k8s_loader.load_pods(object)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/namanjain/Documents/krr/robusta_krr/core/integrations/kubernetes/init.py", line 543, in load_pods
return await cluster_loader.list_pods(object)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/namanjain/Documents/krr/robusta_krr/core/integrations/kubernetes/init.py", line 118, in list_pods
ret: V1PodList = await loop.run_in_executor(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/anaconda3/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/namanjain/Documents/krr/robusta_krr/core/integrations/kubernetes/init.py", line 120, in
lambda: self.core.list_namespaced_pod(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/anaconda3/lib/python3.11/site-packages/kubernetes/client/api/core_v1_api.py", line 15697, in list_namespaced_pod
return self.list_namespaced_pod_with_http_info(namespace, **kwargs) # noqa: E501
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/anaconda3/lib/python3.11/site-packages/kubernetes/client/api/core_v1_api.py", line 15812, in list_namespaced_pod_with_http_info
return self.api_client.call_api(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/anaconda3/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api
return self.__call_api(resource_path, method,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/anaconda3/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
response_data = self.request(
^^^^^^^^^^^^^
File "/usr/local/anaconda3/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request
return self.rest_client.GET(url,
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/anaconda3/lib/python3.11/site-packages/kubernetes/client/rest.py", line 241, in GET
return self.request("GET", url,
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/anaconda3/lib/python3.11/site-packages/kubernetes/client/rest.py", line 235, in request
raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'Audit-Id': '6c010d84-8974-4781-84cb-fd7d95a65e45', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': 'c6fecb2f-6615-45e0-8e84-073e86bb3e81',
'X-Kubernetes-Pf-Prioritylevel-Uid': '6e55d744-0422-4e52-a4ce-de0d56fbdd33', 'Date': 'Wed, 10 Apr 2024 08:53:18 GMT', 'Content-Length': '465'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"unable to parse requirement: values[0][matchLabels]: Invalid value: "{'app':": a valid label must be an empty string or consist of
alphanumeric characters, '-', '' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue', or 'my_value', or '12345', regex used for validation is
'(([A-Za-z0-9][-A-Za-z0-9.])?[A-Za-z0-9])?')","reason":"BadRequest","code":400}
As the error shows that there is some isue with the deployment labels but which deployment it is not showing
since i have 100s of deployment running in my cluster i am not able to find the right one can please help checking this
The text was updated successfully, but these errors were encountered: