Directory sync commands don't set the right ACL to objects #685

anikeev-yadro · 2022-08-26T13:28:02Z

I have tried to use the following commands with parameter "--acl public-read-write":

PS C:\temp> aws --no-verify-ssl s3 cp c:\temp\testdir\d2  s3://b-test-800 --endpoint-url http://172.26.163.38:8084 --acl public-read-write --recursive

PS C:\temp> aws --no-verify-ssl s3 sync c:\temp\testdir\d2  s3://b-test-700 --endpoint-url http://172.26.163.38:8084 --acl public-read-write

with the same result - objects ACL for AllUsers set to public-read instead of public-read-write

PS C:\Users\a.anikeev> aws --no-verify-ssl s3api get-object-acl --bucket b-test-800 --key d1f2.txt --endpoint-url http://172.26.163.38:8084
{
    "Owner": {
        "DisplayName": "NS9iuCpxq8VzpKWLSwceguV5pz2YAYtG6M",
        "ID": "NS9iuCpxq8VzpKWLSwceguV5pz2YAYtG6M"
    },
    "Grants": [
        {
            "Grantee": {
                "ID": "02efc49d370eb40238b85d8469439b5dd70dac7b1567aaf251eef311a312098b13",
                "Type": "CanonicalUser"
            },
            "Permission": "READ"
        },
        {
            "Grantee": {
                "Type": "Group",
                "URI": "http://acs.amazonaws.com/groups/global/AllUsers"
            },
            "Permission": "READ"
        }
    ]
}

Log:

Aug 26 13:16:59 az neofs-s3-gw[5874]: 2022-08-26T13:16:59.959Z        info        api/router.go:167        call method        {"status": 200, "request_id": "509a910b-007b-41ea-b906-7e81f4b82725", "method": "PutObject", "description": "OK"}
Aug 26 13:17:28 az neofs-s3-gw[5874]: 2022-08-26T13:17:28.620Z        error        handler/util.go:25        could not put bucket acl        {"request_id": "f9a5b410-62e3-488c-8d34-dd815bf54d69", "method": "PutObject", "bucket_name": "b-test-800", "object_name": "d2f1.log", "error": "save eACL via connection pool: wait eacl presence on client: context canceled"}
Aug 26 13:17:28 az neofs-s3-gw[5874]: 2022-08-26T13:17:28.620Z        error        api/router.go:158        something went wrong        {"status": 500, "request_id": "f9a5b410-62e3-488c-8d34-dd815bf54d69", "method": "PutObject", "description": "Internal Server Error"}
Aug 26 13:17:28 az neofs-s3-gw[5874]: 2022-08-26T13:17:28.635Z        error        handler/util.go:25        could not put bucket acl        {"request_id": "68f0b520-3279-4e66-a94d-811c46d8b436", "method": "PutObject", "bucket_name": "b-test-800", "object_name": "d2f3.txt", "error": "save eACL via connection pool: wait eacl presence on client: context canceled"}
Aug 26 13:17:28 az neofs-s3-gw[5874]: 2022-08-26T13:17:28.635Z        error        handler/util.go:25        could not put bucket acl        {"request_id": "db63ae59-649a-449f-9cb8-a19089f73eff", "method": "PutObject", "bucket_name": "b-test-800", "object_name": "d2f4.txt", "error": "save eACL via connection pool: wait eacl presence on client: context canceled"}
Aug 26 13:17:28 az neofs-s3-gw[5874]: 2022-08-26T13:17:28.635Z        error        api/router.go:158        something went wrong        {"status": 500, "request_id": "db63ae59-649a-449f-9cb8-a19089f73eff", "method": "PutObject", "description": "Internal Server Error"}
Aug 26 13:17:28 az neofs-s3-gw[5874]: 2022-08-26T13:17:28.635Z        error        api/router.go:158        something went wrong        {"status": 500, "request_id": "68f0b520-3279-4e66-a94d-811c46d8b436", "method": "PutObject", "description": "Internal Server Error"}
Aug 26 13:17:28 az neofs-s3-gw[5874]: 2022-08-26T13:17:28.732Z        error        handler/util.go:25        could not put bucket acl        {"request_id": "2511a93a-79bb-4b62-b676-e9038c38864f", "method": "PutObject", "bucket_name": "b-test-800", "object_name": "test.txt", "error": "save eACL via connection pool: wait eacl presence on client: context canceled"}
Aug 26 13:17:28 az neofs-s3-gw[5874]: 2022-08-26T13:17:28.732Z        error        api/router.go:158        something went wrong        {"status": 500, "request_id": "2511a93a-79bb-4b62-b676-e9038c38864f", "method": "PutObject", "description": "Internal Server Error"}
Aug 26 13:17:30 az neofs-s3-gw[5874]: 2022-08-26T13:17:30.090Z        info        api/router.go:167        call method        {"status": 200, "request_id": "162fefe1-f8ff-4cba-8315-527cd86b7bf0", "method": "PutObject", "description": "OK"}
Aug 26 13:18:00 az neofs-s3-gw[5874]: 2022-08-26T13:18:00.929Z        info        api/router.go:167        call method        {"status": 200, "request_id": "d2d52c15-cab0-4f93-907e-0d3f5a0cfc74", "method": "PutObject", "description": "OK"}
Aug 26 13:18:29 az neofs-s3-gw[5874]: 2022-08-26T13:18:29.399Z        error        handler/util.go:25        could not put bucket acl        {"request_id": "acc557f3-25a2-4a35-a2e6-a379e0ee0b64", "method": "PutObject", "bucket_name": "b-test-800", "object_name": "d2f4.txt", "error": "save eACL via connection pool: wait eacl presence on client: context canceled"}
Aug 26 13:18:29 az neofs-s3-gw[5874]: 2022-08-26T13:18:29.399Z        error        api/router.go:158        something went wrong        {"status": 500, "request_id": "acc557f3-25a2-4a35-a2e6-a379e0ee0b64", "method": "PutObject", "description": "Internal Server Error"}
Aug 26 13:18:29 az neofs-s3-gw[5874]: 2022-08-26T13:18:29.414Z        error        handler/util.go:25        could not put bucket acl        {"request_id": "66f42737-3525-496b-a422-3d36ecb7d6a4", "method": "PutObject", "bucket_name": "b-test-800", "object_name": "d2f3.txt", "error": "save eACL via connection pool: wait eacl presence on client: context canceled"}
Aug 26 13:18:29 az neofs-s3-gw[5874]: 2022-08-26T13:18:29.414Z        error        api/router.go:158        something went wrong        {"status": 500, "request_id": "66f42737-3525-496b-a422-3d36ecb7d6a4", "method": "PutObject", "description": "Internal Server Error"}
Aug 26 13:19:02 az neofs-s3-gw[5874]: 2022-08-26T13:19:02.702Z        info        api/router.go:167        call method        {"status": 200, "request_id": "429a71d2-8c14-4330-b604-660626fe0b7a", "method": "PutObject", "description": "OK"}
Aug 26 13:19:31 az neofs-s3-gw[5874]: 2022-08-26T13:19:31.034Z        error        handler/util.go:25        could not put bucket acl        {"request_id": "5cdb2ea3-f6c3-4728-805b-62743930a175", "method": "PutObject", "bucket_name": "b-test-800", "object_name": "d2f3.txt", "error": "save eACL via connection pool: wait eacl presence on client: context canceled"}
Aug 26 13:19:31 az neofs-s3-gw[5874]: 2022-08-26T13:19:31.034Z        error        api/router.go:158        something went wrong        {"status": 500, "request_id": "5cdb2ea3-f6c3-4728-805b-62743930a175", "method": "PutObject", "description": "Internal Server Error"}

Product versions:

s3 gateway
Version: v0.23.0-36-g3b343d1-dirty
GoVersion: go1.18.4

NeoFS Storage node
Version: v0.31.0
GoVersion: go1.18.4

NeoGo
Version: 0.99.1
GoVersion: go1.18.4

PS C:\Users\a.anikeev> aws --version
aws-cli/2.7.21 Python/3.9.11 Windows/10 exe/AMD64 prompt/off

s3 gateway config:

default_policy: REP 1 IN X CBF 1 SELECT 1 FROM * AS X
listen_address: 0.0.0.0:8084
logger:
  level: debug
max_clients_count: 600
max_clients_deadline: 60s
peers:
  '0':
    address: node1.neofs:8080
    priority: '1'
    weight: '1'
  '1':
    address: node2.neofs:8080
    priority: '2'
    weight: '0.25'
  '2':
    address: node3.neofs:8080
    priority: '2'
    weight: '0.25'
  '3':
    address: node4.neofs:8080
    priority: '2'
    weight: '0.25'
pool_error_threshold: 100
pprof:
  address: localhost:8085
  enabled: true
prometheus:
  address: localhost:8086
  enabled: true
resolve_order:
- nns
rpc_endpoint: http://node1.neofs:40332
tree:
  service: 172.26.163.38:8080
wallet:
  address: ''
  passphrase: ''
  path: /etc/neofs/s3/wallet.json

The text was updated successfully, but these errors were encountered:

KirillovDenis · 2022-08-26T14:55:07Z

Actually READ for object means full control (so just output a little incorrect #677 should fix this issue) because WRITE cannot be applied to object https://docs.aws.amazon.com/AmazonS3/latest/userguide/acl-overview.html#permissions

KirillovDenis · 2022-08-29T12:38:40Z

@anikeev-yadro Could you try to reproduce this bug using ece40d5 commit and see if it's getting better?

anikeev-yadro · 2022-08-30T13:08:47Z

Now it's looks like better.
After we have been uploaded objects with --acl public-read-write:

PS C:\TEMP> aws --no-verify-ssl s3 cp c:\temp\testdir\d2\  s3://b-test-900 --endpoint-url http://172.26.163.38:8084 --acl public-read-write  --metadata m=1 --recursive
upload: testdir\d2\d2f3.txt to s3://b-test-900/d2f3.txt

we see the corresponding ACL:

PS C:\TEMP> aws --no-verify-ssl s3api get-object-acl --bucket b-test-900 --key d2f3.txt --endpoint-url http://172.26.163.38:8084
{
    "Owner": {
        "DisplayName": "NS9iuCpxq8VzpKWLSwceguV5pz2YAYtG6M",
        "ID": "NS9iuCpxq8VzpKWLSwceguV5pz2YAYtG6M"
    },
    "Grants": [
        {
            "Grantee": {
                "ID": "02efc49d370eb40238b85d8469439b5dd70dac7b1567aaf251eef311a312098b13",
                "Type": "CanonicalUser"
            },
            "Permission": "FULL_CONTROL"
        },
        {
            "Grantee": {
                "Type": "Group",
                "URI": "http://acs.amazonaws.com/groups/global/AllUsers"
            },
            "Permission": "FULL_CONTROL"
        }
    ]
}

But I still see errors about ACL in the log:

Aug 30 12:55:43 az neofs-s3-gw[44260]: 2022-08-30T12:55:43.924Z        info        s3-gw/app.go:234        application started        {"name": "neofs-s3-gw", "version": "v0.23.0-51-gece40d5-dirty"}
Aug 30 12:55:43 az neofs-s3-gw[44260]: 2022-08-30T12:55:43.925Z        info        s3-gw/app.go:269        fetch domains, prepare to use API        {"domains": []}
Aug 30 12:55:43 az neofs-s3-gw[44260]: 2022-08-30T12:55:43.927Z        info        s3-gw/app.go:281        starting server        {"bind": "0.0.0.0:8084"}
Aug 30 12:55:43 az neofs-s3-gw[44260]: 2022-08-30T12:55:43.927Z        info        s3-gw/service.go:21        service is running        {"service": "Pprof", "endpoint": "localhost:8085"}
Aug 30 12:55:43 az neofs-s3-gw[44260]: 2022-08-30T12:55:43.927Z        info        s3-gw/service.go:21        service is running        {"service": "Prometheus", "endpoint": "localhost:8086"}
Aug 30 12:58:08 az neofs-s3-gw[44260]: 2022-08-30T12:58:08.948Z        debug        layer/layer.go:371        bucket not found        {"error": "failed resolve: couldn't resolve container 'b-test-900': NNS contract fault exception: at instruction 3437 (THROW): unhandled exception: \"token not found\""}
Aug 30 12:58:17 az neofs-s3-gw[44260]: 2022-08-30T12:58:17.996Z        info        handler/put.go:749        bucket is created        {"container_id": "ExULWGmohPsEpxZdomAA8N2SJ5XugnohSbch9L5TesXw"}
Aug 30 12:58:17 az neofs-s3-gw[44260]: 2022-08-30T12:58:17.996Z        info        api/router.go:167        call method        {"status": 200, "request_id": "8c2cd0ee-a552-4ae0-bb05-e62422021817", "method": "CreateBucket", "description": "OK"}
Aug 30 12:58:48 az neofs-s3-gw[44260]: 2022-08-30T12:58:48.726Z        info        api/router.go:167        call method        {"status": 200, "request_id": "035e2d47-95e9-4bff-af1d-cc354fb80283", "method": "ListObjectsV1", "description": "OK"}
Aug 30 12:59:22 az neofs-s3-gw[44260]: 2022-08-30T12:59:22.907Z        info        api/router.go:167        call method        {"status": 200, "request_id": "1922eb89-71dd-4be3-9576-4f05830ac9e2", "method": "PutObject", "description": "OK"}
Aug 30 12:59:53 az neofs-s3-gw[44260]: 2022-08-30T12:59:53.094Z        info        api/router.go:167        call method        {"status": 200, "request_id": "933ae744-160b-46f0-a359-b93643a3226b", "method": "PutObject", "description": "OK"}
Aug 30 13:00:21 az neofs-s3-gw[44260]: 2022-08-30T13:00:21.792Z        error        handler/util.go:25        could not put bucket acl        {"request_id": "ddd6fdba-940c-4d1f-8e52-47634f29dce3", "method": "PutObject", "bucket_name": "b-test-900", "object_name": "d2f1.log", "error": "save eACL via connection pool: wait eacl presence on client: context canceled"}
Aug 30 13:00:21 az neofs-s3-gw[44260]: 2022-08-30T13:00:21.792Z        error        api/router.go:158        something went wrong        {"status": 500, "request_id": "ddd6fdba-940c-4d1f-8e52-47634f29dce3", "method": "PutObject", "description": "Internal Server Error"}

s3 gate version:

root@az:/usr/bin# /usr/bin/neofs-s3-gw --version
NeoFS S3 Gateway
Version: v0.23.0-51-gece40d5-dirty
GoVersion: go1.18.1

KirillovDenis · 2022-08-30T13:46:25Z

It would be nice to see parameters for authmate command

anikeev-yadro · 2022-08-30T13:57:08Z

authmate command:

anikeev@NB-1670:~/neofs$ sudo ./neofs-s3-authmate-linux-amd64 issue-secret --wallet wallet.json --peer 172.26.163.38:8080 --gate-public-key 02b6c1dc2f13c909918d05e1379f2d684c6fcf668986d199ede10053206acdc4a4 --bearer-rules bearer_rules.json

bearer_rules.json

{
  "records": [
    {"operation": "PUT", "action": "ALLOW", "filters": [], "targets": [{"role": "OTHERS", "keys": []}]},
    {"operation": "GET", "action": "ALLOW", "filters": [], "targets": [{"role": "OTHERS", "keys": []}]},
    {"operation": "HEAD", "action": "ALLOW", "filters": [], "targets": [{"role": "OTHERS", "keys": []}]},
    {"operation": "DELETE", "action": "ALLOW", "filters": [], "targets": [{"role": "OTHERS", "keys": []}]},
    {"operation": "SEARCH", "action": "ALLOW", "filters": [], "targets": [{"role": "OTHERS", "keys": []}]},
    {"operation": "GETRANGE", "action": "ALLOW", "filters": [], "targets": [{"role": "OTHERS", "keys": []}]},
    {"operation": "GETRANGEHASH", "action": "ALLOW", "filters": [], "targets": [{"role": "OTHERS", "keys": []}]}
  ]
}

KirillovDenis · 2022-08-31T13:03:34Z

It seems we cannot do anything with this error:

Aug 30 13:00:21 az neofs-s3-gw[44260]: 2022-08-30T13:00:21.792Z        error        handler/util.go:25        could not put bucket acl        {"request_id": "ddd6fdba-940c-4d1f-8e52-47634f29dce3", "method": "PutObject", "bucket_name": "b-test-900", "object_name": "d2f1.log", "error": "save eACL via connection pool: wait eacl presence on client: context canceled"}

When we create two objects (that require updating EACL) simultaneously, two transactions fall into one block and we can get success result only for one of such EACL update (because we expect eacl table that was sent to be match eacl table that currently can be got).

/cc @alexvanin

alexvanin · 2022-08-31T14:05:27Z

We can do some hacks if requests are sent into the same gateway, e.g. queue AST changes and produce one SetEACL invocation per block. But it is error prone and will not work if requests are sent into two different gateways. But maybe it is good enough for such cases.

alexvanin · 2022-10-03T06:58:44Z

We've decided to propose new mechanism to work with extended ACLs in the container smart contract. Until that we are blocked (or required to build some really dirty fixups in the code).

alexchetaev added U3 Regular and removed 2022Q3 labels Sep 1, 2022

roman-khimov added U4 Nothing urgent S4 Routine I4 No visible changes and removed U3 Regular labels Dec 20, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Directory sync commands don't set the right ACL to objects #685

Directory sync commands don't set the right ACL to objects #685

anikeev-yadro commented Aug 26, 2022

KirillovDenis commented Aug 26, 2022

KirillovDenis commented Aug 29, 2022

anikeev-yadro commented Aug 30, 2022 •

edited

Loading

KirillovDenis commented Aug 30, 2022

anikeev-yadro commented Aug 30, 2022 •

edited

Loading

KirillovDenis commented Aug 31, 2022

alexvanin commented Aug 31, 2022

alexvanin commented Oct 3, 2022

Directory sync commands don't set the right ACL to objects #685

Directory sync commands don't set the right ACL to objects #685

Comments

anikeev-yadro commented Aug 26, 2022

KirillovDenis commented Aug 26, 2022

KirillovDenis commented Aug 29, 2022

anikeev-yadro commented Aug 30, 2022 • edited Loading

KirillovDenis commented Aug 30, 2022

anikeev-yadro commented Aug 30, 2022 • edited Loading

KirillovDenis commented Aug 31, 2022

alexvanin commented Aug 31, 2022

alexvanin commented Oct 3, 2022

anikeev-yadro commented Aug 30, 2022 •

edited

Loading

anikeev-yadro commented Aug 30, 2022 •

edited

Loading