[#136] Bump default tarball_max_size from 8 MiB to 16 MiB

[onno-vos-dev]

I opted for the 2nd option as per described in the issue as it's the quickest and easiest to resolve my issue and allow me to publish docs again. I'll open a separate issue on rebar3_hex to discuss the option of moving to create_docs/2.

[wojtekmach]

Thank you for the PR. This change will up the limit of package tarballs too which I'm not opposed to in general but I'd wait for a concrete use case. If all you need is bigger docs tarball size, I'd rather focus on that.

Changing rebar3_hex is the way to go. In the meantime you can manually publish the docs using this elixir script:

Mix.install([:req])

path = "docs.tar.gz"
{_, 0} = System.cmd("rebar3", ~w(ex_doc), into: IO.stream())
{_, 0} = System.cmd("tar", ~w(-czf #{path} --cd doc .), into: IO.stream())
tarball = File.read!(path)
File.rm_rf!(path)

# $ mix hex.user key generate --permission api:write
token = "..."
package = "hex_core"
version = "0.9.0"

Req.post!("https://hex.pm/api/packages/#{package}/releases/#{version}/docs",
  body: tarball,
  headers: [authorization: token]
)
|> dbg()

It's not a fantastic solution but hopefully it unblocks you in the meantime?

[starbelly]

@wojtekmach What's the limit for doc tarballs on the server side? I believe saw 10MB, can you confirm that is correct?

[onno-vos-dev]

@wojtekmach I can try out the Elixir script you posted later, thanks for that <3

I'd still argue for upping the limit as I'm fairly sure aws_erlang will breach the 8MiB limit fairly soon-ish. It currently sits at around 6.8 MiB and considering that each time AWS opens a new API or expands on existing API's/docs, aws_erlang simply automatically generates the code for it, aws_erlang is essentially only getting bigger and rarely shrinks (AWS rarely deprecates services). So it's mostly a matter of time before we reach the limit.

Is there any chance you can re-consider? :-) Otherwise I'll push for the change in rebar3_hex and try and spend some time on the weekend to make a PR there :-)

[wojtekmach]

I'd still argue for upping the limit as I'm fairly sure aws_erlang will breach the 8MiB limit fairly soon-ish. It currently sits at around 6.8 MiB

oh, ok, thanks for letting us know!

[wojtekmach]

@starbelly yes at the moment hexpm only allows request bodies with ~10mb.

[onno-vos-dev]

@wojtekmach What would be required to have this merged? 😄 Is there anything I can do? 🙂

[ericmj]

I don't think this is the right place for this error since it's error is raised in hex_erl_tarball.

This file is vendored from Erlang's erl_tar module.

[starbelly]

The error should be handled inside rebar3_hex, but I see a gotcha with it, I'll do a fixup there, the error there would look like :

===> Error building package : package exceeds max compressed size 8.388608 MB

[onno-vos-dev]

I will revert the error from this PR 👍

[ericmj]

hex.pm has been updated to allow docs tarballs up to 16MiB size.

[starbelly]

@ericmj To unblock @onno-vos-dev should I just precede with cutting a version of rebar3_hex passing in the new size for docs tarball?

[ericmj]

Closing in favor of #143. Thanks for the PR @onno-vos-dev!