Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement throttling queries to Scylla #67

Open
Bouncheck opened this issue Mar 2, 2022 · 0 comments
Open

Implement throttling queries to Scylla #67

Bouncheck opened this issue Mar 2, 2022 · 0 comments
Labels
enhancement New feature or request

Comments

@Bouncheck
Copy link
Collaborator

Bouncheck commented Mar 2, 2022

It seems that the current behaviour of the connector is to send records as fast as possible, without waiting for futures to complete.
This can lead to issues with some configurations.

For example simple test that sends 10k records failed unexpectedly when ran by github actions runner:

Error:  io.connect.scylladb.integration.ScyllaDbSinkConnectorIT.insert  Time elapsed: 32.609 s  <<< ERROR!
org.apache.kafka.connect.errors.RetriableException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /0.0.0.0:9042 (com.datastax.driver.core.exceptions.BusyPoolException: [/0.0.0.0] Pool is busy (no available connection and the queue has reached its max size 256)))
	at io.connect.scylladb.integration.ScyllaDbSinkConnectorIT.insert(ScyllaDbSinkConnectorIT.java:173)
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /0.0.0.0:9042 (com.datastax.driver.core.exceptions.BusyPoolException: [/0.0.0.0] Pool is busy (no available connection and the queue has reached its max size 256)))
	at io.connect.scylladb.integration.ScyllaDbSinkConnectorIT.insert(ScyllaDbSinkConnectorIT.java:173)
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /0.0.0.0:9042 (com.datastax.driver.core.exceptions.BusyPoolException: [/0.0.0.0] Pool is busy (no available connection and the queue has reached its max size 256)))
	at io.connect.scylladb.integration.ScyllaDbSinkConnectorIT.insert(ScyllaDbSinkConnectorIT.java:173)

This is the method to which the exception points
Looks like it does not care about current number of "in flight" queries before executing another.

@mykaul mykaul added the enhancement New feature or request label Dec 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants