Many small files often have issues downloading

[VideoGamerRUS]

Bug description

Whose problem it is? Server, my or Automodpack?
I`m trying to resolve this timeout moments

[01:52:38] [AutoModpackDownload-1/WARN] [AutoModpack/]: Timeout - automodpack\modpacks\ERC_SMP10\tacz\tacz_default_gun\assets\tacz\tacz_sounds\p90\p90_inspect_grabgrip.ogg - java.net.SocketTimeoutException: Connect timed out - [java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:551), java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:602), java.base/java.net.Socket.connect(Socket.java:633), java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:178), java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:533), java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:638), java.base/sun.net.www.http.HttpClient.(HttpClient.java:281), java.base/sun.net.www.http.HttpClient.New(HttpClient.java:386), java.base/sun.net.www.http.HttpClient.New(HttpClient.java:408), java.base/sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1309), java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1242), java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1128), java.base/sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1057), java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1665), java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1589), LAYER SERVICE/[email protected]/pl.skidam.automodpack_loader_core.utils.DownloadManager.downloadFile(DownloadManager.java:159), LAYER SERVICE/[email protected]/pl.skidam.automodpack_loader_core.utils.DownloadManager.downloadTask(DownloadManager.java:56), LAYER SERVICE/[email protected]/pl.skidam.automodpack_loader_core.utils.DownloadManager.lambda$downloadNext$0(DownloadManager.java:121), java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804), java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136), java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635), java.base/java.lang.Thread.run(Thread.java:833)]
[01:52:38] [AutoModpackDownload-1/WARN] [AutoModpack/]: Download of p90_inspect_grabgrip.ogg failed, retrying!

One message in the server console:
[01:51:05 WARN]: An exception was thrown by pl.skidam.automodpack_core.netty.HttpServerHandler$$Lambda/0x000001ab4efb5ba8.operationComplete()
java.lang.NullPointerException: Cannot invoke "io.netty.channel.ChannelHandlerContext.channel()" because the return value of "io.netty.channel.ChannelPipeline.firstContext()" is null
at pl.skidam.automodpack_core.netty.HttpServerHandler.lambda$sendFile$0(HttpServerHandler.java:151) ~[automodpack-mc1.20.1-forge-4.0.0-beta22.jar%23425!/:?]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578) ~[netty-common-4.1.82.Final.jar%23174!/:4.1.82.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:552) ~[netty-common-4.1.82.Final.jar%23174!/:4.1.82.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491) ~[netty-common-4.1.82.Final.jar%23174!/:4.1.82.Final]
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616) ~[netty-common-4.1.82.Final.jar%23174!/:4.1.82.Final]
at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:605) ~[netty-common-4.1.82.Final.jar%23174!/:4.1.82.Final]
at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104) ~[netty-common-4.1.82.Final.jar%23174!/:4.1.82.Final]
at io.netty.util.internal.PromiseNotificationUtil.trySuccess(PromiseNotificationUtil.java:48) ~[netty-common-4.1.82.Final.jar%23174!/:4.1.82.Final]
at io.netty.channel.ChannelOutboundBuffer.safeSuccess(ChannelOutboundBuffer.java:717) ~[netty-transport-4.1.82.Final.jar%23177!/:4.1.82.Final]
at io.netty.channel.ChannelOutboundBuffer.remove(ChannelOutboundBuffer.java:272) ~[netty-transport-4.1.82.Final.jar%23177!/:4.1.82.Final]
at io.netty.channel.nio.AbstractNioByteChannel.doWriteInternal(AbstractNioByteChannel.java:242) ~[netty-transport-4.1.82.Final.jar%23177!/:4.1.82.Final]
at io.netty.channel.nio.AbstractNioByteChannel.doWrite0(AbstractNioByteChannel.java:212) ~[netty-transport-4.1.82.Final.jar%23177!/:4.1.82.Final]
at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:407) ~[netty-transport-4.1.82.Final.jar%23177!/:4.1.82.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:931) ~[netty-transport-4.1.82.Final.jar%23177!/:4.1.82.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.forceFlush(AbstractNioChannel.java:361) ~[netty-transport-4.1.82.Final.jar%23177!/:4.1.82.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:782) ~[netty-transport-4.1.82.Final.jar%23177!/:4.1.82.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[netty-transport-4.1.82.Final.jar%23177!/:4.1.82.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[netty-transport-4.1.82.Final.jar%23177!/:4.1.82.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[netty-transport-4.1.82.Final.jar%23177!/:4.1.82.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.82.Final.jar%23174!/:4.1.82.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.82.Final.jar%23174!/:4.1.82.Final]
at java.lang.Thread.run(Thread.java:1570) [?:?]

Steps to reproduce

1. Add to your modpack 1000 min-mid size files (emotes from emotecraft or addons for TacZ)
2. Start downloading the modpack

Expected behavior

No response

Actual behavior

No response

Relevant logs

https://mclo.gs/twvl0QK - full client log

Minecraft & Mod Loader versions

1.20.1 Forge 47.3.11

Minecraft launcher

Prism Launcher

Operating system

Win 10

AutoModpack version

beta 22

Other information

After every 5-10 files a timeout

Check list

• I have verified that the issue persists in the latest version of the mod.
• I have searched the existing issues and confirmed that this is not a duplicate.

[Skidamek]

Hi, could you try dev build from github actions?

[VideoGamerRUS]

Hi, could you try dev build from github actions?

I`ve tryied the build (server+client)

Here are the client logs: https://mclo.gs/vre6mmF

This i had on the server console:
[19:00:21 ERROR]: Error writing to channel: java.io.IOException: Connection reset by peer path: automodpack\host-modpack\main\emotes\SPE_Running with swords.json

[Skidamek]

Oh.. i dont know, it more looks like more like bandwidth/server can't keep up with sending you all of modpack files. Not sure how to fix it. Could you just use smaller modpack? lol

[VideoGamerRUS]

Oh.. i dont know, it more looks like more like bandwidth/server can't keep up with sending you all of modpack files. Not sure how to fix it. Could you just use smaller modpack? lol

Yes, I use the smaller one. I just want players to have everything at once without having to download the emotes separately. Could this be a problem because the files are so small?

[Skidamek]

Idk, it surely could be somehow fixed in automodpack, i don't know underlying cause of this

[Skidamek]

you may try beta 23, but likely it wont fix it

[VideoGamerRUS]

you may try beta 23, but likely it wont fix it

It looks to be fixed in the dev build used in #325

[Skidamek]

Really? well, that's awesome lol

[VideoGamerRUS]

Really? well, that's awesome lol

Im doing more tests right now, and will feedback soon

[VideoGamerRUS]

Really? well, that's awesome lol

Im doing more tests right now, and will feedback soon

Well. It is much better, but still happens sometimes.
40 mins downloading --> 10 mins 👍

[Skidamek]

Please try latest build from github actions
builds download - its big, since i have added few dependencies and these are 18 builds for 18 versions/loaders lol

[VideoGamerRUS]

Please try latest build from github actions builds download - its big, since i have added few dependencies and these are 18 builds for 18 versions/loaders lol

https://mclo.gs/lLZ17tE it is impossible to connect

[Skidamek]

Hmm, do you use some forge bukkit engine mix?

[VideoGamerRUS]

Hmm, do you use some forge bukkit engine mix?

Yup. MohistMC

[Skidamek]

It breaks because of this org.bukkit.plugin.SimplePluginManager.callEvent(SimplePluginManager.java:583)

[VideoGamerRUS]

It breaks because of this org.bukkit.plugin.SimplePluginManager.callEvent(SimplePluginManager.java:583)

I need to report this to Mohist dev, yes?

[Skidamek]

Yeah, you can report. Do you have installed SimplePluginManager plugin or is it something coming from mohist? I will try to figure it out in some free time

[VideoGamerRUS]

Yeah, you can report. Do you have installed SimplePluginManager plugin or is it something coming from mohist? I will try to figure it out in some free time

Nah. Mohist core component, i think.
Well, i'll open a bug report for Mohist

[Skidamek]

Try new build, now I call checkCanJoin method (canPlayerLogin in mojmap) only on server thread (that's how I am actually supposed to do lol) and this should fix your issue.

However mohist may want to move its PlayerLoginEvent fire somewhere else or add some additional check to it since automodpack needs to use this method even if player is nowhere near logging to minecraft server - that may mess with plugins using this event.

[VideoGamerRUS]

Try new build, now I call checkCanJoin method (canPlayerLogin in mojmap) only on server thread (that's how I am actually supposed to do lol) and this should fix your issue.

However mohist may want to move its PlayerLoginEvent fire somewhere else or add some additional check to it since automodpack needs to use this method even if player is nowhere near logging to minecraft server - that may mess with plugins using this event.

Tested. Now is generating another error, triggering other methods. so, im waiting the answer from Mohist

[Skidamek]

Tested. Now is generating another error, triggering other methods. so, im waiting the answer from Mohist

Fixed in automodpack, build

I've searched for use of this method using grep.app and i found only one meaningful way of usage that would actually affect my modpack authorization check #324, and it is BanHammer, however calling that method would also cause banhammer to break so for it to work it would require changes in banhammer itself.

So, i came to a conclusion that writing simpler method myself makes more sense, since this way, it won't break anything but only may not respect custom ban/whitelist mods/plugins like banhammer but thats a smaller problem.

[VideoGamerRUS]

Tested. Now is generating another error, triggering other methods. so, im waiting the answer from Mohist

Fixed in automodpack, build

I've searched for use of this method using grep.app and i found only one meaningful way of usage that would actually affect my modpack authorization check #324, and it is BanHammer, however calling that method would also cause banhammer to break so for it to work it would require changes in banhammer itself.

So, i came to a conclusion that writing simpler method myself makes more sense, since this way, it won't break anything but only may not respect custom ban/whitelist mods/plugins like banhammer but thats a smaller problem.

The build works and works greate, no problems with big modpacks, but in my opinion, is very necessary respect the whitelist mods/plugins. Can we wait the answer from Mohist? Because I really don't want this to be cut out. Its not a small problem, we need this feature(

[Skidamek]

Not sure about that one, on grep I found few mods which completely don't account for a fact that this method could be potentially called in some other point than on player login itself breaking themselves (bukkit event fire from this point also broke luckperms as I've seen from your mohist report), it's not a good method to use in general.

Anyway let's go back to this issue topic, does small files still causes problems? Is download faster or slower? How does it feel like?

[VideoGamerRUS]

Not sure about that one, on grep I found few mods which completely don't account for a fact that this method could be potentially called in some other point than on player login itself breaking themselves (bukkit event fire from this point also broke luckperms as I've seen from your mohist report), it's not a good method to use in general.

Anyway let's go back to this issue topic, does small files still causes problems? Is download faster or slower? How does it feel like?

Well. I hope it will have a resolve.. About this topic: 1733 files downloaded +- in 1 minute with no issues. You did a greate work!! I think we're done here! Thanks!

[Skidamek]

Well. It is much better, but still happens sometimes. 40 mins downloading --> 10 mins 👍

Are these the same files you were downloading 40 mins before?

[VideoGamerRUS]

Well. It is much better, but still happens sometimes. 40 mins downloading --> 10 mins 👍

Are these the same files you were downloading 40 mins before?

Yup. The same emotes for Emotecraft
UPD for the whitelist: It doesn't break him, and all works correct! <3

[Skidamek]

It's crazy, I am impressed...

[Skidamek]

I wouldn't close this issue yet, since I got different report that it's still slightly broken - some files just don't download, if you want, you can try to stresstest it few days and let me know if something breaks on your end as well

[VideoGamerRUS]

I wouldn't close this issue yet, since I got different report that it's still slightly broken - some files just don't download, if you want, you can try to stresstest it few days and let me know if something breaks on your end as well

Ok, i`ll do! If i find something, will feedback