Exporters should use queue jobs

[modelesque]

Description

My client needs to export all of their orders every week. This is because our orders can be placed up to a year or two in advance, and they go through several different order statuses with lots of things changing throughout that process until the order is delivered.

I built a custom export for my client to gather only the fields needed per order, but now the export is timing out (with about 1000 orders currently in the system). I wish I could do this via the CLI but it's something I need our accounting department to be able to do on their own time via the CMS.

So far it doesn't appear possible to export data batched into background queue jobs so that it doesn't time out in the browser. Or is it possible? If so, is there documentation on it? If not, will this be coming in a new release in the very near future?

I imagine this to be a common problem for sites growing with hundreds or thousands of entries/users/orders/etc. What is the Craft way to handle this?

Steps to reproduce

1. Generate a CMS instance with 1000 elements of your choice.
2. Export the data on one of the listing pages with those elements.

Additional info

• Craft version: 3.6.16
• PHP version: 7.3.27
• Database driver & version: MySQL
• Plugins & versions: Commerce 3.3.3

[brandonkelly]

It wouldn’t be possible to move the export functionality to a queue job without adding some complexity to the workflow, such as adding a new section for viewing/downloading your recent exports, once available. So I’ve moved this to a discussion.

[modelesque]

I see. I suppose the way it works now is fine for rather limited lists of elements like certain channels of Entrys, but what about lists of elements that can grow indefinitely for a site like Users and Orders? I think ultimately, it would be nice if the CP could handle exporting any of its data regardless of length. Right now I have to write a workaround to run the export via CLI and email the exported CSV to my client. It just feels unnecessarily cumbersome just to get my data out of the system. Just my 2¢.

[khalwat]

I ran into this with @modelesque -- the problem is that the exports are done via frontend web request.

At some point, you will accrue more Commerce Orders (or whatever you're exporting) than can be processed in the timeout from a frontend web request.

In this case, it's only 1,000 orders. Then it times out, and the export fails.

I get that this would require an architectural change, so probably isn't a quick fix. But I would almost consider this more of a bug, and less of a feature request... only because it makes the basic export capability have a cap that can render it useless.

[brandonkelly]

Partial exports (via checkboxes) work fine regardless of the number of total elements. And while yes the timeout itself is a bug, there’s no way to fix it directly given the constraints of the way it works currently. So if you want it to be fixed, we need to approach it as an enhancement (beefing up the export workflow), that just so happens to fix a bug along the way.