You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This is reproduce-able on k8s deployments, where runs vitess-operator with vitess v16.
When consolidator been enabled, run select query concurrently at large scale through vtgate, and vttablet container get OOMKilled.
set relatively small memory for vttablet container (limit at 1Gi for example)
craft a select query, and make the size of returned relatively large (5Mi for example)
run above select query concurrently at large scale through vtgate (10,000 queries for example)
observe vttablet OOMKilled
Binary Version
Vitess 16 and after versions.
/vt/bin$ ./vtgate --version
Version: 16.0.3-SNAPSHOT (Git revision 4335eaf8ce3fa328aacd36e66f4776bd5208c7c8 branch 'v16-hc-demonware') built on Tue Dec 12 18:02:03 UTC 2023 by vitess@buildkitsandbox using go1.20.5 linux/amd64
/vt/bin$ ./vttablet --version
Version: 16.0.3-SNAPSHOT (Git revision 4335eaf8ce3fa328aacd36e66f4776bd5208c7c8 branch 'v16-hc-demonware') built on Tue Dec 12 18:02:03 UTC 2023 by vitess@buildkitsandbox using go1.20.5 linux/amd64
Operating System and Environment details
kubernetes version: v1.27.11
Log Fragments
OOMKilled happens very quick before any log can be outputted.
The text was updated successfully, but these errors were encountered:
@jwangace thank you for the report! Seeing that v16 is unsupported, could you please clarify whether the bug still appeas on supported versions (v19, v20, v21 at this time)?
Hi @shlomi-noach as you might have noticed, I also put a fix proposal PR in the latest code, unfortunately because we don't have any v22 deployments so I did not reproduce that on v22, however I cross compared related function (in which I proposed to update execSelect) and I believe this bug should present up to the current.
Do you think this is something PlantScale can verify by following Reproduction Steps?
Overview of the Issue
This is reproduce-able on k8s deployments, where runs vitess-operator with vitess v16.
When consolidator been enabled, run select query concurrently at large scale through vtgate, and vttablet container get OOMKilled.
Consolidated Query Wait Count (vttablet_waits_count)
![Screenshot 2024-11-16 at 7 59 54 AM](https://private-user-images.githubusercontent.com/121262788/386860096-9f7e9036-83fe-46e0-9233-4fdcc740c0bd.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzQzNzc1MTUsIm5iZiI6MTczNDM3NzIxNSwicGF0aCI6Ii8xMjEyNjI3ODgvMzg2ODYwMDk2LTlmN2U5MDM2LTgzZmUtNDZlMC05MjMzLTRmZGNjNzQwYzBiZC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQxMjE2JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MTIxNlQxOTI2NTVaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0wZDg4NTQ1NjE3ODA2MzUzYjEyYzNlN2I2ZWYyNmJjNGM3ZjdlMTlkZTFmZjIwMzhmOWEwZjE3NzY2MzQwMmVlJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9.B4rVXcbiRkjDJgpWYf1GkhSJ0KTN85TamjGaj2esaeI)
OOMKilled Metrics
![Screenshot 2024-11-16 at 7 58 15 AM](https://private-user-images.githubusercontent.com/121262788/386859782-300b10c0-db38-4009-abb0-1479c990c7f6.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzQzNzc1MTUsIm5iZiI6MTczNDM3NzIxNSwicGF0aCI6Ii8xMjEyNjI3ODgvMzg2ODU5NzgyLTMwMGIxMGMwLWRiMzgtNDAwOS1hYmIwLTE0NzljOTkwYzdmNi5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQxMjE2JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MTIxNlQxOTI2NTVaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT05NGQxODQxYjRhMmJkOGViYWI5ZmMxODNkYWFkZmJlYzAyYzUzMThkYzVkZjc1MjgwYmEyZGYzYzcxOTYxMzZlJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9.pjY1fJW9A1k7HhQLvCdSygyROsQBT2SXCN_i1BpHBi8)
Reproduction Steps
To easier reproduce this issue, you can:
Binary Version
Vitess 16 and after versions.
Operating System and Environment details
Log Fragments
The text was updated successfully, but these errors were encountered: