-
Notifications
You must be signed in to change notification settings - Fork 44
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add a field to limit the size of uploading content #1701
base: main
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Was the goal to read blob data in smaller chunks and in case we are blocking the API for too long, we raise an error?
pulp_container/pulp_container/app/registry_api.py
Lines 1353 to 1360 in 25abfd6
while True: | |
subchunk = chunk.read(2000000) | |
if not subchunk: | |
break | |
temp_file.write(subchunk) | |
size += len(subchunk) | |
for algorithm in Artifact.DIGEST_FIELDS: | |
hashers[algorithm].update(subchunk) |
I see a threshold defined for config blobs: https://github.com/containers/image/blob/1dbd8fbbe51653e8a304122804431b07a1060d06/internal/image/oci.go#L63-L83. But, I could not find any thresholds for regular blobs. Can we investigate this?
@@ -938,6 +942,10 @@ def put(self, request, path, pk=None): | |||
repository.pending_blobs.add(blob) | |||
return BlobResponse(blob, path, 201, request) | |||
|
|||
def _verify_payload_size(self, distribution, chunk): | |||
if distribution.max_payload_size and chunk.reader.length > distribution.max_payload_size: | |||
raise PayloadTooLarge() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this exception properly rendered to the container's API format?
ref:
def handle_exception(self, exc): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some of the exceptions are properly documented at https://docker-docs.uclv.cu/registry/spec/api/#blob-upload.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ahn... my bad, I didn't know about the spec for errors.
I also found this doc https://github.com/opencontainers/distribution-spec/blob/v1.0.1/spec.md#error-codes that will be helpful.
@@ -751,6 +751,8 @@ class ContainerDistribution(Distribution, AutoAddObjPermsMixin): | |||
null=True, | |||
) | |||
|
|||
max_payload_size = models.IntegerField(null=True) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I do not think we want to make this parameter configurable. I vote for having a constant for this.
Hum... when I was writing this PR I was thinking about a feature to allow a maximum size for blob layers and deny it if the limit is exceeded.
I also couldn't find a limit for regular blobs. From what I could understand, they limit only the size of buffered "resources" (mainifests, config blobs, signatures, etc.) to avoid OOM issues: |
Okay, I think we can follow that path. Let's then focus on manifests, config blobs, and signatures exclusively. |
After an internal discussion, here is the conclusion we got:
|
This change makes sense to me, however besides manifest directive we need to also cap Config blobs are left out but i don't see an easy way to set 4mb limit specifically on them given that they are in One thing I do not understand, if the malicious user can create big manifest that would lead to big memory consumption, what prevents him to just create big blob? There is no limit set on the blobs. |
Skimming through commit description containers/image@61096ab changes, they are mostly targeted on the client side, so the client during pull operation is not susceptible to DDoS attack. |
Hm, apparently the changes were applied also on some upload calls like https://github.com/containers/image/blob/61096ab72530cc9216b50d9fc3bee90b53704f68/docker/docker_image_dest.go#L630 |
closes: #532