You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The docs, and previous experience, are leading users to still generating csv.gz outputs, even when using ehrql. This is expensive in terms of limited server CPU.
We should consider making this to be an error in ehrql, which exits with a message to say to use arrow. However, we should probably provide an --allow-csv flag to make it possible if needed.
This may need some coordination, as the docs need updating, and we need to make sure there's a good solution to viewing arrow files in local development
The text was updated successfully, but these errors were encountered:
Just copying some thoughts from a Slack thread so they're salient when we next come to look at this:
I'm fully behind the goal here, but I'm not sure adding flags like this is the way to do it. For one thing, users have exhibited a strong tendency to copy/paste stuff wholesale from previous projects. If someone adds the --allow-csv flag somewhere ("just to get things working") then it can start making it's way into other projects without those users ever explicitly thinking about it.
In the first instance, I'd like to start by exploiting the fact that we have all our users' code available to us and searchable and do some regular (monthly?) check for people using CSV inappropriately.
We've also previously discussed having some more generally "opensafely lint" step which could check things like output formats (alongside lots of other stuff).
I wouldn't want to leap in to adding weird shit to the ehrql interface as our first port of call.
The docs, and previous experience, are leading users to still generating csv.gz outputs, even when using ehrql. This is expensive in terms of limited server CPU.
We should consider making this to be an error in ehrql, which exits with a message to say to use arrow. However, we should probably provide an --allow-csv flag to make it possible if needed.
This may need some coordination, as the docs need updating, and we need to make sure there's a good solution to viewing arrow files in local development
The text was updated successfully, but these errors were encountered: