Export hangs and doesn't continue

[shootie22]

Version

2.39.1

Flavor

GUI (Graphical User Interface)

Export format

HTML

Details

Hi DCE devs, I want to first of all thank you for this amazing project. You are doing god's work.

I'm having an issue with exporting chats, it seems to hang after a while and I cannot figure out when or what causes it. What I did was to put in my token, CTRL + A on my DMs and export as HTML to a folder on my Desktop. My C drive has enough storage space. It exports fine until the very last bit, where it just hangs and doesn't continue:

I tried exporting the chats from a small server, and it hung a bit after starting:

I can't figure out what is happening so I'm filing this bug report to hopefully get it working. Thanks in advance!

Steps to reproduce

1. Paste in your auth code
2. CTRL + A to select all chats
3. Export as HTML, dark, into a folder
4. Export

[shootie22]

uh oh I forgot to censor my auth token 💀

[Yudi]

uh oh I forgot to censor my auth token 💀

Your token will reset if you change your password

[shootie22]

fixed, thanks for the tip

[Tyrrrz]

Can you test if it also happens on v2.36.4?

[shootie22]

yup, v2.36.4 seems not to have that issue. managed to export all my DMs without issues

[Tyrrrz]

yup, v2.36.4 seems not to have that issue. managed to export all my DMs without issues

Are you using a bot or a user token?

[shootie22]

I'm using a user token

[Tyrrrz]

So it seems that it might be caused by new rate limit handling (introduced in v2.37, right after v2.36.4). However, the delays should be capped, meaning that the export should still progress further at some point. If it gets fully stuck, then the issue might be something else.

[Tyrrrz]

Could also be caused by #1103. Does the channel contain many server invites?

[decoherencer]

So it seems that it might be caused by new rate limit handling (introduced in v2.37, right after v2.36.4). However, the delays should be capped, meaning that the export should still progress further at some point. If it gets fully stuck, then the issue might be something else.

Is there a way to force the delay to resync, or get a visual signal when delay is active on next expected time? It is stuck for hours and I have no idea if it is working or not,
and if I start again for such stuck guild, I have to do from start again i guess?

[Tyrrrz]

Is there a way to force the delay to resync, or get a visual signal when delay is active on next expected time? It is stuck for hours and I have no idea if it is working or not,
and if I start again for such stuck guild, I have to do from start again i guess?

The delay should never exceed one minute between requests, AFAIK. So if it's stuck for hours, something else is going wrong. I asked someone to test an older version and it seems they experienced slow exports there too, so I'm not sure what's wrong.

[nightbug08]

I am also appearing to have this problem. It downloaded about a third of the channels I set it to, then the yellow loading bar just hung there. Then it started getting smaller whenever I clicked another server icon? It's not even giving me 0kb ghost files for anything now, even on another laptop 20 hours later.

I can still chat in all my servers and DM people though, so it's confusing.

[quartztester]

@Tyrrrz I am able to reproduce this error on v2.42.6 using a user auth token. Similar to @shootie22 this is through exporting DMs and I don't believe #1103 is the issue.

Is there any way to check the status of the downloads other than the visual indicator in the application window? Are there any other diagnostic criteria I should look into?

Looking at my windows task manager there's 0 Network usage leading to the conclusion that the program is stuck in some regard (after a solid 30 minutes of waiting).

[Tyrrrz]

Is there any way to check the status of the downloads other than the visual indicator in the application window? Are there any other diagnostic criteria I should look into?

There is no logging, but you might get some insight but looking at the export file and seeing which message it gets stuck on. If it's consistently getting stuck at the same place, it would imply the issue is in the content that gets exported. If it gets stuck at random places, then the issue is probably somewhere else.

In any case, I'd also ask to test in v2.36.4 as well to rule out the rate-limiting behavior I mentioned earlier in the comments.

[quartztester]

It has been consistently getting stuck in a different place from the few servers that I've tried. Issue is most likely somewhere else. v2.36.4 was successful in server and DM exports.

If you'd like me to perform any other test let me know @Tyrrrz.

[quartztester]

Side note, what update was thread support added?

[Tyrrrz]

v2.36.4 was successful in server and DM exports.

Okay then it's most likely caused by the new rate-limiting behavior.

Side note, what update was thread support added?

Depends on in which capacity you mean. Because threads are just regular channels, so exporting them was always possible (in CLI).

[quartztester]

Is there any way I can test the or diagnose the rate-limiting behavior?
Why was it added in the first place? Could there be an added option to change the behavior of the rate limit in future updates?

Depends on in which capacity you mean.

In regards to GUI support as threads aren't scraped in v2.36.4 and there doesn't seem to be a mention of threads in the changelog other than fixes to other thread related fixes.

[281732]

Having the same issue using the latest docker image, works fine on version 2.36.2

[Tyrrrz]

Can you guys try this CI build?
https://github.com/Tyrrrz/DiscordChatExporter/actions/runs/7160184410?pr=1171

[cheese529]

I have been running into this same issue for the last 48 hours. Exports keep getting stuck especially in channels with a lot of invalid server invite links. Is there anyway at all to fix this? Tried with both the latest version of version 2.38

[Tyrrrz]

I have been running into this same issue for the last 48 hours. Exports keep getting stuck especially in channels with a lot of invalid server invite links. Is there anyway at all to fix this? Tried with both the latest version of version 2.38

Have you tried the version linked above?

[cheese529]

I have been running into this same issue for the last 48 hours. Exports keep getting stuck especially in channels with a lot of invalid server invite links. Is there anyway at all to fix this? Tried with both the latest version of version 2.38

Have you tried the version linked above?

Will try it tonight after work and let you know how it works. FYI after switching accounts and IP address, it started working again on version 2.38.

[cheese529]

UPDATE: Tried the version linked above and it does seem to work better.

[Tyrrrz]

UPDATE: Tried the version linked above and it does seem to work better.

Try using it for a while and see if it works better consistently. This issue seems to be non-deterministic, so it's better to gather a larger sample size.

[cheese529]

i'll get right on it, going to try exporting channels ranging from 20k messages to over 500k messages with invite links as well.

[traxart79]

im also having this issue with the latest version, anyone got fixes?

[quartztester]

Can you guys try this CI build? https://github.com/Tyrrrz/DiscordChatExporter/actions/runs/7160184410?pr=1171

@Tyrrrz Are you able to re-open the pull request? I never got an opportunity to download the test build.

[quartztester]

I'm currently am using v2.43.3 (Windows x64) and am not receiving the same issue. I am exporting with these options:

JSON,
No Date/Partition/Filter,
Format markdown - FALSE,
Download assets - TRUE,
Reuse assets - TRUE
Assets directory path - Default,

Show Threads - All
Parallel limit - 2

[shootie22]

I'm still experiencing this issue on the latest (v2.43.3) on Linux desktop x64. It still hangs just like in the initial screenshot posted. It seems like it doesn't completely stop, but it just starts going very slowly, so it's possible it gets rate limited for some reason. I think this is the case because if I check the output folder, new files are very slowly being created still (about 1-2 files/s).

HTML Dark
Show Threads - All
Download assets - True
Reuse assets - True
Assets directory path - Default
Parallel limit - Max

[quartztester]

@shootie22 Try to decrease your parallel limit, I think Discord has a way to rate throttle when automation is detected. While that might not be the exact cause it'd help to diagnose.

[Tyrrrz]

Reviving an old thread, but if I remember correctly through my testing the conclusion was that:

1. Discord was indeed throttling user accounts
2. The throttling was unpredictably applied
3. It was soft-throttling, i.e. the headers in the last response indicated that the client should wait X seconds before proceeding with the next request, but that delay was not actually enforced by Discord
4. Previously, DCE did not respect rate limits with user accounts as Discord did not enforce them (except for extremely rare cases, in which a retry-immediately-after was sufficient)

So because of that our solutions are limited to:

1. Don't respect rate limits with user accounts (revert back to old behavior). In practice, it was fine before, but some users were concerned that they might get banned. Also, if you download your Discord data, the logs would show you exceeding rate limits on certain occasions.
2. Add a setting that controls whether to respect rate limits. Essentially a more flexible version of the previous solution.
3. Have a fixed rate limit of our own for user accounts that doesn't follow Discord's response headers. In theory, there should be a way to find a good balance between hitting rate limit warnings in log and having to wait too long. But ultimately this would be a volatile solution based on assumptions.

[quartztester]

@Tyrrrz Yes, agreeing with your analysis

[Tyrrrz]

I implemented solution 2 (a setting to control whether to respect rate limits) in #1342. Please test the build from that PR to see if it makes a difference.