Loki - input size too long

[jbartyze-rh]

Hello everyone,

I am wondering if there is a workaround for current Loki limitation that is present in the stack or is there anything planned for the future to circumvent that?

Problem:
If user sees too many namespaces, query will be blocked by Loki.

For user that does not possess cluster-admin they may possess view of all namespaces in form of cluster-monitoring-view or cluster-reader
For user that does possess cluster-admin opreator errors out if character count is above 5120 and is not stable for global views.

Example environment:

OCP 4.12 cluster on vSphere, 10 operators installed with namespaces character count above 5120 Loki query limit.

Error as below.

[jotak]

Hi @jbartyze-rh

Unfortunately for non-cluster admins, we don't have a good solution/workaround at the moment for this problem.
We plan working on a workaround by supporting a new feature of the Loki Operator related the namespace-based permissions: https://github.com/aminesnow/enhancements/blob/2a7aa2ae2139b5eaec6a3f623cc2a82f4ea2071e/enhancements/cluster-logging/fine-grained-per-namespace-logs-access.md

When we will implement it, the workaround when you hit this issue will be to create RoleBindings per namespace (instead of ClusterRoleBinding) for the non-admin users with Read access to logs, with the intent to reduce the number of accessible namespaces in that way. It will not eliminate the problem if you keep assigning as many namespaces, but it offers you the possibility to have less namespaces just for netobserv flow logs access.

Would that be an acceptable trade-off for you?

[jotak]

FYI I just created a ticket to better track that: https://issues.redhat.com/browse/NETOBSERV-1324

[jbartyze-rh]

Hi @jotak thx for checking my comment! Sorry for late response, was on PTO. Did some more research on that issue, please correct me if I am wrong or there is some logic I missed!

So basically based on https://access.redhat.com/solutions/7018952 cluster-admin user need to be added to cluster-admin group. For it to circumvent that limitation of Loki 5120 query size. Customer would not be happy; they already use Groups for specific platform teams with CRB assigned to it and those groups are synced from AD by LdapSync cronjob.

So for example platform team having assigned the cluster-admin role to platform-admin Group.

Creating another group or another Role to circumvent arbitrary limitation on Loki seems like a little overkill. I imagine there is a specific reason why we went this direction. What Customer would want is to define 1 group for particular group of people and assign CRB or RB to it through GitOps to give necessary permissions on the cluster.

for non-cluster admins - we can only restrict the view with an enhancement. Sadly, I do not think that would cover use cases where we have 1 line support teams or external vendors that have cluster-reader access on the clusters and should see everything to troubleshoot their issues. I would argue such teams would want ability to query all namespaces and would want to see all namespaces network metrics from network-observability operator.

Maybe instead of having cluster-admin group, we could use some different logic here to validate if user should have ability to query more than 5120 characters.

In comparison Elasticsearch did not have such restrictions in our previous releases of our logging stack and kibana users could create index-patterns to query through all index-patterns if they wanted.

[jotak]

@jbartyze-rh I get your point & frustration - this would be something to discuss with openshift-logging team as they maintain the loki-operator.

[jbartyze-rh]

No worries 👍 will try to bring that conversation to them so we make both products better! I wanted to get your opinion first here as we observe this issue in Network Observability Operator.

[jotak]

By the way, can you elaborate on that please?

For user that does possess cluster-admin opreator errors out if character count is above 5120 and is not stable for global views.

cluster-admins should not get this error. Just note that they need to be in the cluster-admin group, not just the role : cf https://issues.redhat.com/browse/LOG-4236?focusedId=22459085&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-22459085

[jbartyze-rh]

Did not know about that workaround! Thx! Will test that one :)

[jbartyze-rh]

Just a quick question, is that cluster-admin group pattern used anywhere else or it was added for this particular use-case?

[jotak]

I don't know all the background behind this design choice: this is actually in the loki-operator and not netobserv itself .. sorry!