Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Complex object is completely unreadable after upgrade to SDK RC-9 #2448

Closed
cthulhu-rider opened this issue Jul 20, 2023 · 5 comments
Closed
Assignees
Labels
bug Something isn't working neofs-storage Storage node application issues
Milestone

Comments

@cthulhu-rider
Copy link
Contributor

i made same actions but with later NeoFS revision, but the result is completely different

all nodes respond with

rpc error: read object header: status: code = 2049 message = object not found

log:

2023-07-20T16:51:36.059Z	debug	get/get.go:87	serving request...	{"component": "Object.Get service", "request": "GET", "address": "EPQ4rJVAvY8qsgRGiu3nVhGNJuCe1hEPnCfxRB4BhgdC/DtWBKhHDFkb6Qd8wiZ54j4prTRjBoPrkMXcEGf5rUXei", "raw": false, "local": false, "with session": false, "with bearer": false}
2023-07-20T16:51:36.059Z	warn	engine/engine.go:142	could not get object from shard	{"shard_id": "HbtWhXBvdEofwDxLQhaFou", "error": "no split info on parent object"}
2023-07-20T16:51:36.059Z	debug	get/local.go:25	local get failed	{"component": "Object.Get service", "request": "GET", "address": "EPQ4rJVAvY8qsgRGiu3nVhGNJuCe1hEPnCfxRB4BhgdC/DtWBKhHDFkb6Qd8wiZ54j4prTRjBoPrkMXcEGf5rUXei", "raw": false, "local": false, "with session": false, "with bearer": false, "error": "status: code = 2049 message = object not found"}
2023-07-20T16:51:36.059Z	debug	get/get.go:108	operation finished with error	{"component": "Object.Get service", "request": "GET", "address": "EPQ4rJVAvY8qsgRGiu3nVhGNJuCe1hEPnCfxRB4BhgdC/DtWBKhHDFkb6Qd8wiZ54j4prTRjBoPrkMXcEGf5rUXei", "raw": false, "local": false, "with session": false, "with bearer": false, "error": "status: code = 2049 message = object not found"}
2023-07-20T16:51:36.059Z	debug	get/container.go:18	trying to execute in container...	{"component": "Object.Get service", "request": "GET", "address": "EPQ4rJVAvY8qsgRGiu3nVhGNJuCe1hEPnCfxRB4BhgdC/DtWBKhHDFkb6Qd8wiZ54j4prTRjBoPrkMXcEGf5rUXei", "raw": false, "local": false, "with session": false, "with bearer": false, "netmap lookup depth": 0}
2023-07-20T16:51:36.059Z	debug	get/container.go:46	process epoch	{"component": "Object.Get service", "request": "GET", "address": "EPQ4rJVAvY8qsgRGiu3nVhGNJuCe1hEPnCfxRB4BhgdC/DtWBKhHDFkb6Qd8wiZ54j4prTRjBoPrkMXcEGf5rUXei", "raw": false, "local": false, "with session": false, "with bearer": false, "number": 2}
2023-07-20T16:51:36.059Z	debug	get/remote.go:14	processing node...	{"component": "Object.Get service", "request": "GET", "address": "EPQ4rJVAvY8qsgRGiu3nVhGNJuCe1hEPnCfxRB4BhgdC/DtWBKhHDFkb6Qd8wiZ54j4prTRjBoPrkMXcEGf5rUXei", "raw": false, "local": false, "with session": false, "with bearer": false}
2023-07-20T16:51:36.062Z	debug	get/remote.go:34	remote call failed	{"component": "Object.Get service", "request": "GET", "address": "EPQ4rJVAvY8qsgRGiu3nVhGNJuCe1hEPnCfxRB4BhgdC/DtWBKhHDFkb6Qd8wiZ54j4prTRjBoPrkMXcEGf5rUXei", "raw": false, "local": false, "with session": false, "with bearer": false, "error": "status: code = 2049 message = object not found"}
2023-07-20T16:51:36.062Z	debug	get/remote.go:14	processing node...	{"component": "Object.Get service", "request": "GET", "address": "EPQ4rJVAvY8qsgRGiu3nVhGNJuCe1hEPnCfxRB4BhgdC/DtWBKhHDFkb6Qd8wiZ54j4prTRjBoPrkMXcEGf5rUXei", "raw": false, "local": false, "with session": false, "with bearer": false}
2023-07-20T16:51:36.065Z	debug	get/remote.go:34	remote call failed	{"component": "Object.Get service", "request": "GET", "address": "EPQ4rJVAvY8qsgRGiu3nVhGNJuCe1hEPnCfxRB4BhgdC/DtWBKhHDFkb6Qd8wiZ54j4prTRjBoPrkMXcEGf5rUXei", "raw": false, "local": false, "with session": false, "with bearer": false, "error": "status: code = 2049 message = object not found"}
2023-07-20T16:51:36.065Z	debug	get/container.go:63	no more nodes, abort placement iteration	{"component": "Object.Get service", "request": "GET", "address": "EPQ4rJVAvY8qsgRGiu3nVhGNJuCe1hEPnCfxRB4BhgdC/DtWBKhHDFkb6Qd8wiZ54j4prTRjBoPrkMXcEGf5rUXei", "raw": false, "local": false, "with session": false, "with bearer": false}
2023-07-20T16:51:36.065Z	debug	get/get.go:108	operation finished with error	{"component": "Object.Get service", "request": "GET", "address": "EPQ4rJVAvY8qsgRGiu3nVhGNJuCe1hEPnCfxRB4BhgdC/DtWBKhHDFkb6Qd8wiZ54j4prTRjBoPrkMXcEGf5rUXei", "raw": false, "local": false, "with session": false, "with bearer": false, "error": "status: code = 2049 message = object not found"}

as opposed to access problem from another issue, here node can't even reach the fact that requested object is complex (virtual). I guess the problem may be in status codes and/or SDK client errors

Versions

  • neofs-cli v0.37.0
  • neofs-node >= 18e3f68106f4e9d5f5ecda8f4b1caef74358be55 (not yet released)
@cthulhu-rider cthulhu-rider added bug Something isn't working triage neofs-storage Storage node application issues labels Jul 20, 2023
@roman-khimov roman-khimov added this to the v0.38.0 milestone Jul 20, 2023
@cthulhu-rider
Copy link
Contributor Author

i have not seen no split info on parent object error in log before, can be caused by problems with slicing? @carpawell

@cthulhu-rider
Copy link
Contributor Author

i guess i found the reason: parent header contains split ID (slicing mistake) while it shouldn't. It's not obvious why metabase fails due to this, need to explore separately. If node considers such objects as problem, then it must deny them on PUT stage. I'll create an issue

@roman-khimov
Copy link
Member

Do I understand correctly that we can't release node with SDK RC9 and we definitely need RC10 for everything to work correctly?

@roman-khimov
Copy link
Member

Fixed by #2470?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working neofs-storage Storage node application issues
Projects
None yet
Development

No branches or pull requests

2 participants