Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

miq orchestrator container doesn't propagate right environment variables to workers #20582

Closed
p-v-a opened this issue Sep 23, 2020 · 3 comments
Closed
Assignees
Labels

Comments

@p-v-a
Copy link

p-v-a commented Sep 23, 2020

I'm running jansa-1 version on vanilla k8s 1.17.2. I have several issues with miq orchestrator container either not passing required information into worker containers or MIQ ignores it

  1. Secrets names are hardcoded - postgresql-secrets, kafka-secrets, app-secrets are hardcoded in MIQ and as far as I can see there is no way to overwrite it see - databaseSecret not propagated to the workers manageiq-pods#601

  2. Even with correct secrets worker pods in a crashloop state
    NAME READY STATUS RESTARTS AGE
    1-event-handler-6cc5bfff7c-7rj96 1/1 Running 5 4m24s
    1-generic-54855c57fd-c76sx 0/1 CrashLoopBackOff 3 3m46s
    1-generic-54855c57fd-mv2fp 0/1 CrashLoopBackOff 4 3m46s
    1-priority-5994c67b7d-s7jrs 0/1 CrashLoopBackOff 4 4m24s
    1-priority-5994c67b7d-zpscq 1/1 Running 4 4m24s
    1-remote-console-745d976499-8krqg 0/1 Error 5 4m25s
    1-reporting-6fd59749c6-4l9k5 0/1 CrashLoopBackOff 4 3m27s
    1-reporting-6fd59749c6-zmjsc 0/1 CrashLoopBackOff 3 3m27s
    1-schedule-5fb84c567b-264f2 0/1 CrashLoopBackOff 4 3m48s
    1-ui-7d9dcd76db-zfdxr 0/1 Running 1 27s
    1-web-service-677b86f9f7-w28l9 0/1 CrashLoopBackOff 1 100s
    miq-dev01-memcached-667dfb8767-xzgn2 1/1 Running 0 150m
    miq-dev01-orchestrator-58dc4b567-gg755 1/1 Running 0 123m
    miq-dev01-postgresql-deployment-7497f5d5c7-rn5z4 1/1 Running 0 161m

with the errors

{"@timestamp":"2020-09-23T05:26:03.425134 ","hostname":"1-ui-7d9dcd76db-l8d7z","pid":8,"tid":"2ad73e4f195c","level":"info","message":"MIQ(SessionStore) Using session_store: ActionDispatch::Session::MemCacheStore"}
{"@timestamp":"2020-09-23T05:26:03.885417 ","hostname":"1-ui-7d9dcd76db-l8d7z","pid":8,"tid":"2ad73e4f195c","level":"debug","message":"Dalli::Server#connect 127.0.0.1:11211"}
{"@timestamp":"2020-09-23T05:26:03.886121 ","hostname":"1-ui-7d9dcd76db-l8d7z","pid":8,"tid":"2ad73e4f195c","level":"warning","message":"127.0.0.1:11211 failed (count: 0) Errno::ECONNREFUSED: Connection refused - connect(2) for "127.0.0.1" port 11211"}

Looks like it defaults to local memcached if no environment variables are provided. I think orchestrator passes MEMCACHED_SERVER and MEMCACHED_SERVICE_NAME into workers. Yet orchestrator itself expects memcached connection details in MEMCACHED_SERVICE_HOST and MEMCACHED_SERVICE_PORT.
Even if I add MEMCACHED_SERVER and MEMCACHED_SERVICE_NAME to the pod definition it looks like it never used, I see same error in the logs.

@p-v-a
Copy link
Author

p-v-a commented Sep 23, 2020

All worked pods were started successfully after I patched worker deployment definitions by adding MEMCACHED_SERVER and MEMCACHED_SERVICE_NAME variables.

@p-v-a
Copy link
Author

p-v-a commented Sep 23, 2020

Sorry I confused myself. I have added MEMCACHED_SERVICE_HOST and MEMCACHED_SERVICE_PORT, not MEMCACHED_SERVER and MEMCACHED_SERVICE_NAME variables I mentioned above.

@gtanzillo gtanzillo added bug and removed question labels Sep 24, 2020
@chessbyte
Copy link
Member

@bdunne @abellotti is this still an issue in Kasparov release?

@p-v-a p-v-a closed this as completed Jun 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants