Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add checks for lifetime, expire_at flags #621

Merged
merged 7 commits into from
Sep 1, 2023

Conversation

evgeniiz321
Copy link
Contributor

@evgeniiz321 evgeniiz321 commented Aug 30, 2023

Currently there are multiple problems here (allure with logs is attached):

  1. New test test_bearer_token_expiration (previously bearer token was not created by cli command but was created manually, now I use 'bearer create' command). 'get range hash' command doesn't work with the generated token, it fails like this:
COMMAND: neofs-cli --config /home/runner/work/neofs-node/neofs-node/neofs-testcases/wallet_config.yml object hash --rpc-endpoint 's01.neofs.devenv:8080' --wallet '/home/runner/work/neofs-node/neofs-node/neofs-testcases/TemporaryDir/e768c827-812f-4a8c-9090-c52a3b7047ed.json' --cid 'HMVBh53JLaPF38Ztca3TQKfgjPwvFhBto1bSeREjfyiy' --oid '27wMEkckZ2VUAEtu3bLWymTabDn7Rw6BANvKss5jwt2r' --bearer '/home/runner/work/neofs-node/neofs-node/neofs-testcases/TemporaryDir/TestFilesDir/bearer_token_e95282b4-3ab9-4db3-8f86-1dd8a64694d6' --range '0:10'
RETCODE: 1

STDOUT:
rpc error: read payload hashes via client: status: code = 2049 message = object not found

STDERR:

Start / End / Elapsed	 20:52:08.234392 / 20:52:08.751591 / 0:00:00.517199

In the logs there are the following entries:

2023-08-30T20:51:54.306Z	debug	get/get.go:87	serving request...	{"component": "Object.Get service", "request": "GET_RANGE", "address": "AT95KqvYRw3AC1cCmPJdxwYAcXDJGFLv89rZaZKsmJk3/CdrUYtHAuDDzFF8iw4mAgN2qqb8SDKPo8Gpyg12Ree2k", "raw": false, "local": false, "with session": false, "with bearer": true}
2023-08-30T20:51:54.306Z	debug	get/local.go:25	local get failed	{"component": "Object.Get service", "request": "GET_RANGE", "address": "AT95KqvYRw3AC1cCmPJdxwYAcXDJGFLv89rZaZKsmJk3/CdrUYtHAuDDzFF8iw4mAgN2qqb8SDKPo8Gpyg12Ree2k", "raw": false, "local": false, "with session": false, "with bearer": true, "error": "status: code = 2049 message = object not found"}
2023-08-30T20:51:54.306Z	debug	get/get.go:108	operation finished with error	{"component": "Object.Get service", "request": "GET_RANGE", "address": "AT95KqvYRw3AC1cCmPJdxwYAcXDJGFLv89rZaZKsmJk3/CdrUYtHAuDDzFF8iw4mAgN2qqb8SDKPo8Gpyg12Ree2k", "raw": false, "local": false, "with session": false, "with bearer": true, "error": "status: code = 2049 message = object not found"}
2023-08-30T20:51:54.306Z	debug	get/container.go:18	trying to execute in container...	{"component": "Object.Get service", "request": "GET_RANGE", "address": "AT95KqvYRw3AC1cCmPJdxwYAcXDJGFLv89rZaZKsmJk3/CdrUYtHAuDDzFF8iw4mAgN2qqb8SDKPo8Gpyg12Ree2k", "raw": false, "local": false, "with session": false, "with bearer": true, "netmap lookup depth": 0}
2023-08-30T20:51:54.306Z	debug	get/container.go:46	process epoch	{"component": "Object.Get service", "request": "GET_RANGE", "address": "AT95KqvYRw3AC1cCmPJdxwYAcXDJGFLv89rZaZKsmJk3/CdrUYtHAuDDzFF8iw4mAgN2qqb8SDKPo8Gpyg12Ree2k", "raw": false, "local": false, "with session": false, "with bearer": true, "number": 7}
2023-08-30T20:51:54.306Z	debug	get/remote.go:13	processing node...	{"component": "Object.Get service", "request": "GET_RANGE", "address": "AT95KqvYRw3AC1cCmPJdxwYAcXDJGFLv89rZaZKsmJk3/CdrUYtHAuDDzFF8iw4mAgN2qqb8SDKPo8Gpyg12Ree2k", "raw": false, "local": false, "with session": false, "with bearer": true}
2023-08-30T20:51:54.311Z	debug	get/container.go:87	completing the operation	{"component": "Object.Get service", "request": "GET_RANGE", "address": "AT95KqvYRw3AC1cCmPJdxwYAcXDJGFLv89rZaZKsmJk3/CdrUYtHAuDDzFF8iw4mAgN2qqb8SDKPo8Gpyg12Ree2k", "raw": false, "local": false, "with session": false, "with bearer": true}
2023-08-30T20:51:54.311Z	debug	get/get.go:99	operation finished successfully	{"component": "Object.Get service", "request": "GET_RANGE", "address": "AT95KqvYRw3AC1cCmPJdxwYAcXDJGFLv89rZaZKsmJk3/CdrUYtHAuDDzFF8iw4mAgN2qqb8SDKPo8Gpyg12Ree2k", "raw": false, "local": false, "with session": false, "with bearer": true}
2023-08-30T20:51:54.787Z	debug	get/get.go:87	serving request...	{"component": "Object.Get service", "request": "GET_RANGE", "address": "AT95KqvYRw3AC1cCmPJdxwYAcXDJGFLv89rZaZKsmJk3/CdrUYtHAuDDzFF8iw4mAgN2qqb8SDKPo8Gpyg12Ree2k", "raw": false, "local": false, "with session": false, "with bearer": true}
2023-08-30T20:51:54.787Z	debug	get/local.go:25	local get failed	{"component": "Object.Get service", "request": "GET_RANGE", "address": "AT95KqvYRw3AC1cCmPJdxwYAcXDJGFLv89rZaZKsmJk3/CdrUYtHAuDDzFF8iw4mAgN2qqb8SDKPo8Gpyg12Ree2k", "raw": false, "local": false, "with session": false, "with bearer": true, "error": "status: code = 2049 message = object not found"}
2023-08-30T20:51:54.787Z	debug	get/get.go:108	operation finished with error	{"component": "Object.Get service", "request": "GET_RANGE", "address": "AT95KqvYRw3AC1cCmPJdxwYAcXDJGFLv89rZaZKsmJk3/CdrUYtHAuDDzFF8iw4mAgN2qqb8SDKPo8Gpyg12Ree2k", "raw": false, "local": false, "with session": false, "with bearer": true, "error": "status: code = 2049 message = object not found"}
2023-08-30T20:51:54.787Z	debug	get/container.go:18	trying to execute in container...	{"component": "Object.Get service", "request": "GET_RANGE", "address": "AT95KqvYRw3AC1cCmPJdxwYAcXDJGFLv89rZaZKsmJk3/CdrUYtHAuDDzFF8iw4mAgN2qqb8SDKPo8Gpyg12Ree2k", "raw": false, "local": false, "with session": false, "with bearer": true, "netmap lookup depth": 0}
2023-08-30T20:51:54.787Z	debug	get/container.go:46	process epoch	{"component": "Object.Get service", "request": "GET_RANGE", "address": "AT95KqvYRw3AC1cCmPJdxwYAcXDJGFLv89rZaZKsmJk3/CdrUYtHAuDDzFF8iw4mAgN2qqb8SDKPo8Gpyg12Ree2k", "raw": false, "local": false, "with session": false, "with bearer": true, "number": 7}
2023-08-30T20:51:54.787Z	debug	get/remote.go:13	processing node...	{"component": "Object.Get service", "request": "GET_RANGE", "address": "AT95KqvYRw3AC1cCmPJdxwYAcXDJGFLv89rZaZKsmJk3/CdrUYtHAuDDzFF8iw4mAgN2qqb8SDKPo8Gpyg12Ree2k", "raw": false, "local": false, "with session": false, "with bearer": true}
2023-08-30T20:51:54.792Z	debug	get/remote.go:29	remote call failed	{"component": "Object.Get service", "request": "GET_RANGE", "address": "AT95KqvYRw3AC1cCmPJdxwYAcXDJGFLv89rZaZKsmJk3/CdrUYtHAuDDzFF8iw4mAgN2qqb8SDKPo8Gpyg12Ree2k", "raw": false, "local": false, "with session": false, "with bearer": true, "error": "init object reading: header: status: code = 2048 message = access to object operation denied"}
2023-08-30T20:51:54.792Z	debug	get/remote.go:13	processing node...	{"component": "Object.Get service", "request": "GET_RANGE", "address": "AT95KqvYRw3AC1cCmPJdxwYAcXDJGFLv89rZaZKsmJk3/CdrUYtHAuDDzFF8iw4mAgN2qqb8SDKPo8Gpyg12Ree2k", "raw": false, "local": false, "with session": false, "with bearer": true}
2023-08-30T20:51:54.826Z	debug	get/remote.go:29	remote call failed	{"component": "Object.Get service", "request": "GET_RANGE", "address": "AT95KqvYRw3AC1cCmPJdxwYAcXDJGFLv89rZaZKsmJk3/CdrUYtHAuDDzFF8iw4mAgN2qqb8SDKPo8Gpyg12Ree2k", "raw": false, "local": false, "with session": false, "with bearer": true, "error": "init object reading: header: status: code = 2048 message = access to object operation denied"}
2023-08-30T20:51:54.826Z	debug	get/remote.go:13	processing node...	{"component": "Object.Get service", "request": "GET_RANGE", "address": "AT95KqvYRw3AC1cCmPJdxwYAcXDJGFLv89rZaZKsmJk3/CdrUYtHAuDDzFF8iw4mAgN2qqb8SDKPo8Gpyg12Ree2k", "raw": false, "local": false, "with session": false, "with bearer": true}
2023-08-30T20:51:54.836Z	debug	get/remote.go:29	remote call failed	{"component": "Object.Get service", "request": "GET_RANGE", "address": "AT95KqvYRw3AC1cCmPJdxwYAcXDJGFLv89rZaZKsmJk3/CdrUYtHAuDDzFF8iw4mAgN2qqb8SDKPo8Gpyg12Ree2k", "raw": false, "local": false, "with session": false, "with bearer": true, "error": "init object reading: header: status: code = 2048 message = access to object operation denied"}
2023-08-30T20:51:54.836Z	debug	get/container.go:63	no more nodes, abort placement iteration	{"component": "Object.Get service", "request": "GET_RANGE", "address": "AT95KqvYRw3AC1cCmPJdxwYAcXDJGFLv89rZaZKsmJk3/CdrUYtHAuDDzFF8iw4mAgN2qqb8SDKPo8Gpyg12Ree2k", "raw": false, "local": false, "with session": false, "with bearer": true}
2023-08-30T20:51:54.836Z	debug	get/get.go:108	operation finished with error	{"component": "Object.Get service", "request": "GET_RANGE", "address": "AT95KqvYRw3AC1cCmPJdxwYAcXDJGFLv89rZaZKsmJk3/CdrUYtHAuDDzFF8iw4mAgN2qqb8SDKPo8Gpyg12Ree2k", "raw": false, "local": false, "with session": false, "with bearer": true, "error": "status: code = 2049 message = object not found"}

All other commands work as expected. It only happens on get range hash

  1. Same test test_bearer_token_expiration, I set lifetime=1 and expire_at=current_epoch + 1 and I expect that the token will expire after 2 epoch ticks. In other scenarios it works ok. But here, I have to set expire_at=current_epoch + 2 during the token creation to have the same behavior as with lifetime=1. It seems strange.

  2. Another new test - test_session_token_expiration_flags, seems like expire-at param for a session token doesn't have any effect. Lifetime works ok.

All failures are in the attached allure - allure.tar.gz or here - https://github.com/evgeniiz321/neofs-node/suites/15662779232/artifacts/894066888

@vvarg229
Copy link
Collaborator

Doesn't this PR: nspcc-dev/neofs-sdk-go#503 fix this problem?

@roman-khimov
Copy link
Member

Doesn't this PR: nspcc-dev/neofs-sdk-go#503 fix this problem?

It's a different thing, here we add some tests for CLI utilities/network logic around expirations. That PR improves session token handling in SDK to solve spurious gateway error problem.

Copy link
Member

@roman-khimov roman-khimov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

'get range hash' command doesn't work with the generated token

It's a protocol issue. Please add it to https://github.com/nspcc-dev/neofs-api/ and please add getrange to the token, getrangehash can't work on its own and won't do that.

I have to set expire_at=current_epoch + 2 during the token creation to have the same behavior as with lifetime=1

This needs to be checked at the node side (please add an issue), if it can be reproduced on every run then the test should be disabled for now.

test_session_token_expiration_flags, seems like expire-at param for a session token doesn't have any effect. Lifetime works ok

Please add a node issue and disable the test temporarily.

Evgeniy Zayats added 2 commits August 31, 2023 21:22
Evgeniy Zayats added 3 commits August 31, 2023 21:54
Test to validate lifetime, expire_at params

Signed-off-by: Evgeniy Zayats <[email protected]>
Add proper checks for lifetime, expire_at params

Signed-off-by: Evgeniy Zayats <[email protected]>
Add proper checks for lifetime, expire_at params

Signed-off-by: Evgeniy Zayats <[email protected]>
@evgeniiz321 evgeniiz321 changed the title Add checks for lifetime, expire_at flags Add checks for lifetime, expire_at flags, closes #612 Sep 1, 2023
@evgeniiz321 evgeniiz321 changed the title Add checks for lifetime, expire_at flags, closes #612 Add checks for lifetime, expire_at flags Sep 1, 2023
@evgeniiz321
Copy link
Contributor Author

evgeniiz321 commented Sep 1, 2023

Added issues, updated commits, disabled failing tests.
Fresh allure with all latest updates - allure-6.tar.gz

Didn't get the part with: please add getrange to the token, getrangehash can't work on its own and won't do thatAdd it to the ACL? getrange is there, as well as all other operations.

To validate session token expiration flags - lifetime, expire_at

Signed-off-by: Evgeniy Zayats <[email protected]>
Signed-off-by: Evgeniy Zayats <[email protected]>
@evgeniiz321
Copy link
Contributor Author

evgeniiz321 commented Sep 1, 2023

One more thing - in my opinion CLI tests that check just CLI code (e.g. that lifetime and expire-at flags are mutually exclusive, or the generation of a bearer token) should be done in other type of tests, more low level (component). Tests here are too expensive to do checks like these. We can't add them here as much as we can - eventually we will be out of resources in any case. So I suggest to start adding more tests in PRs like this - nspcc-dev/neofs-node#2521, written in go and with the whole system mocked, just to validate that CLI logic works.

@roman-khimov
Copy link
Member

getrange is there, as well as all other operations.

IIUC your token has one operation enabled with all others disabled. A token with getrangehash only is dysfunctional as you've noticed already and the reason is that you can't have getrangehash working with getrange disabled, so if you're to add both to the token it'll work.

@evgeniiz321
Copy link
Contributor Author

evgeniiz321 commented Sep 1, 2023

No, token should has all operations enabled, it was generated with the following acl:

COMMAND: neofs-cli --config /home/runner/work/neofs-node/neofs-node/neofs-testcases/wallet_config.yml acl extended create --cid 'HMVBh53JLaPF38Ztca3TQKfgjPwvFhBto1bSeREjfyiy' --out '/home/runner/work/neofs-node/neofs-node/neofs-testcases/TemporaryDir/TestFilesDir/eacl_table_82a063fd-c6af-4a9c-905f-979d1de560ec.json' --rule 'allow put  user' --rule 'allow get  user' --rule 'allow head  user' --rule 'allow getrange  user' --rule 'allow getrangehash  user' --rule 'allow search  user' --rule 'allow delete  user'
RETCODE: 0

STDOUT:

STDERR:

Start / End / Elapsed	 20:52:03.805656 / 20:52:03.819742 / 0:00:00.014086

@roman-khimov
Copy link
Member

Huh. Then it's a different bug, please add to neofs-node repo.

@roman-khimov roman-khimov merged commit e95c173 into nspcc-dev:master Sep 1, 2023
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants