Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add wait flag to rollout command #15

Merged
merged 8 commits into from
Feb 19, 2021
Merged

Conversation

jtrouth
Copy link
Contributor

@jtrouth jtrouth commented Feb 16, 2021

This should resolve the transient connection issues for SQL proxy and RabbitMQ on deploy

Kim Scott and others added 7 commits December 21, 2020 10:31
update cloudsql, google app, aws and sendgrid keys, and secret key fo…
update google cloud sql for prod
update google/cloudsql, S3, sendgrid, & postgres credentials for prod
update secret key for prod
@jtrouth
Copy link
Contributor Author

jtrouth commented Feb 16, 2021

Resolves #5 & #6

@kimberscott kimberscott changed the base branch from master to develop February 19, 2021 18:16
@kimberscott
Copy link

Thanks! Trying this out on a build now.

@kimberscott kimberscott merged commit 472ad33 into lookit:develop Feb 19, 2021
@kimberscott
Copy link

Ok, I kicked off a fresh lookit-api staging build using a lookit-orchestrator build with this fix at 1:20pm today. Would you be able to look at the logs and let me know if the transient connection errors still happened? I'm having trouble finding the relevant logs in GCP's new logs explorer.

@jtrouth
Copy link
Contributor Author

jtrouth commented Feb 24, 2021

Still seeing the errors. You can see the errors with the beat pod connecting to sqlproxy with the query severity=ERROR resource.type="k8s_container" textPayload:"OperationalError"

I think the solution is going to end up being to add init containers to the pods that wait for the sqlproxy or rabbitmq services to start responding. I'll do some local testing to verify init containers are the solution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants