readiness probes of solr pod with probesRequireAuth set to true always return "Success" regardless of the actual result #799
Description
The Readiness Probes of the chart always return success as the command of the authenticated wget is always piped into a grep command which always has the return code 0 regardless weather they were able to access the Endpoint or not. This leads to the solr-pod starting clean and kept alive by k8s even though the authentication to the endpoints is actually failing. In my opinion this shall lead to the pod not becoming ready.
The same applies to the Liveness Probes respectively.
the generated yaml of the probes from the solr pod by the solr-operator:
livenessProbe:
exec:
command:
- sh
- -c
- JAVA_TOOL_OPTIONS="-Dbasicauth=$(cat /etc/secrets/solr-k8s-oper-credentials/username):$(cat
/etc/secrets/solr-k8s-oper-credentials/password) -Dsolr.httpclient.builder.factory=org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory"
solr api -get "http://${SOLR_HOST}:8983/solr/admin/info/system" 2>&1 | grep
-v JAVA_TOOL_OPTIONS
failureThreshold: 3
periodSeconds: 20
successThreshold: 1
timeoutSeconds: 5
name: solrcloud-node
ports:
- containerPort: 8983
name: solr-client
protocol: TCP
readinessProbe:
exec:
command:
- sh
- -c
- JAVA_TOOL_OPTIONS="-Dbasicauth=$(cat /etc/secrets/solr-k8s-oper-credentials/username):$(cat /etc/secrets/solr-k8s-oper-credentials/password) -Dsolr.httpclient.builder.factory=org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory" solr api -get "http://${SOLR_HOST}:8983/solr/admin/info/health" 2>&1 | grep -v JAVA_TOOL_OPTIONSsolr@solr-solrcloud-0:/opt/solr-9.2.1$ JAVA_TOOL_OPTIONS="-Dbasicauth=$(cat /etc/secrets/solr-k8s-oper-credentials/username):$(cat /etc/secrets/solr-k8s-oper-credentials/password) -Dsolr.httpclient.builder.factory=org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory" solr api -get "http://${SOLR_HOST}:8983/solr/admin/info/health" 2>&1 | grep -v JAVA_TOOL_OPTIONS
ERROR: Parse error : <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 401 Bad credentials</title>
</head>
<body><h2>HTTP ERROR 401 Bad credentials</h2>
<table>
<tr><th>URI:</th><td>/solr/admin/info/health</td></tr>
<tr><th>STATUS:</th><td>401</td></tr>
<tr><th>MESSAGE:</th><td>Bad credentials</td></tr>
<tr><th>SERVLET:</th><td>default</td></tr>
</table>
</body>
</html>
solr@solr-solrcloud-0:/opt/solr-9.2.1$ echo $?
0
solr@solr-solrcloud-0:/opt/solr-9.2.1$ JAVA_TOOL_OPTIONS="-Dbasicauth=$(cat /etc/secrets/solr-k8s-oper-credentials/username):$(cat /etc/secrets/solr-k8s-oper-credentials/password) -Dsolr.httpclient.builder.factory=org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory" solr api -get "http://${SOLR_HOST}:8983/solr/admin/info/health" 2>&1
Picked up JAVA_TOOL_OPTIONS: -Dbasicauth=k8s-oper:***REDACTED*** -Dsolr.httpclient.builder.factory=org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory
ERROR: Parse error : <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 401 Bad credentials</title>
</head>
<body><h2>HTTP ERROR 401 Bad credentials</h2>
<table>
<tr><th>URI:</th><td>/solr/admin/info/health</td></tr>
<tr><th>STATUS:</th><td>401</td></tr>
<tr><th>MESSAGE:</th><td>Bad credentials</td></tr>
<tr><th>SERVLET:</th><td>default</td></tr>
</table>
</body>
</html>
solr@solr-solrcloud-0:/opt/solr-9.2.1$ echo $?
1
Downside is that without the grep -v the basicAuth Password may be logged. I can't figure out where it may be logged but, I assume the grep -v may have been introduce to suppress the logging of the credentials.
supplying the correct basic auth also renders the correct return code (only without the return code - suppressing grep):
solr@solr-solrcloud-0:/opt/solr-9.2.1$ JAVA_TOOL_OPTIONS="-Dbasicauth=$(cat /etc/secrets/solr-k8s-oper-credentials/username):$(cat /etc/secrets/solr-k8s-oper-credentials/password) -Dsolr.httpclient.builder.factory=org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory" solr api -get "http://${SOLR_HOST}:8983/solr/admin/info/health" 2>&1
Picked up JAVA_TOOL_OPTIONS: -Dbasicauth=k8s-oper:***REDACTED*** -Dsolr.httpclient.builder.factory=org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory
{
"responseHeader":{
"status":0,
"QTime":0},
"status":"OK"}
solr@solr-solrcloud-0:/opt/solr-9.2.1$ echo $?
0