Query method not working for large files

[jiexixi]

Documentation feedback

• Page title: Query CrateDB
• Page URL: https://crate.io/docs/python/en/latest/query.html
• Source: https://github.com/crate/crate-python/blob/master/docs/query.rst

---

[jiexixi]

Hi there, I tried to query some large files on crateDB with the above method but it threw the error message

crate.client.exceptions.ProgrammingError: CircuitBreakingException[[query] Data too large, data for [http-result] would be [645922816/616mb], which is larger than the limit of [644245094/614.3mb]]

It seems that the file has exceeded the size limit of the function. Could you please recommend a way to query the large files (around 18GB)? Thank you.

[proddata]

Hi @jiexixi

To retrieve large resultsets (i.e. larger than what would fit in the heap) you need to either use a client / driver that supports a fetch size like asyncpg https://github.com/MagicStack/asyncpg or JDBC on a protocol level or run multiple sub-queries and merge the result on the client.