Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Resizing a volume used external metadata fails even though the storage pool has enough free space #360

Open
kanno2inf opened this issue Jun 14, 2023 · 0 comments · May be fixed by #367
Open

Resizing a volume used external metadata fails even though the storage pool has enough free space #360

kanno2inf opened this issue Jun 14, 2023 · 0 comments · May be fixed by #367

Comments

@kanno2inf
Copy link

Growing volume fails due to metadata pool capacity exceeded even though the storage pool has enough free space.

Error report

$ linstor error-reports show 64895E66-00000-000000
ERROR REPORT 64895E66-00000-000000

============================================================

Application:                        LINBIT® LINSTOR
Module:                             Controller
Version:                            1.23.0
Build ID:                           28dbd33ced60d75a2a0562bf5e9bc6b800ae8361
Build time:                         2023-05-23T07:06:58+00:00
Error time:                         2023-06-14 15:30:38
Node:                               sn1
Peer:                               RestClient(0:0:0:0:0:0:0:1; 'PythonLinstor/1.18.0 (API1.0.4): Client 1.18.0')

============================================================

Reported error:
===============

Category:                           RuntimeException
Class name:                         ApiRcException
Class canonical name:               com.linbit.linstor.core.apicallhandler.response.ApiRcException
Generated at:                       Method 'ensureAllStorPoolsHaveEnoughFreeSpace', Source file 'CtrlVlmDfnModifyApiCallHandler.java', Line #671

Error message:                      Cannot grow the volume definition by 4091904KiB, as the following storage pool do not have enough free space:
Node: sn1, StorPool: meta
Node: sn2, StorPool: meta

Error context:
    Cannot grow the volume definition by 4091904KiB, as the following storage pool do not have enough free space:
    Node: sn1, StorPool: meta
    Node: sn2, StorPool: meta

Asynchronous stage backtrace:

    Error has been observed at the following site(s):
        |_ checkpoint ⇢ Modify volume definition
    Stack trace:

Call backtrace:

    Method                                   Native Class:Line number
    ensureAllStorPoolsHaveEnoughFreeSpace    N      com.linbit.linstor.core.apicallhandler.controller.CtrlVlmDfnModifyApiCallHandler:671

Suppressed exception 1 of 1:
===============
Category:                           RuntimeException
Class name:                         OnAssemblyException
Class canonical name:               reactor.core.publisher.FluxOnAssembly.OnAssemblyException
Generated at:                       Method 'ensureAllStorPoolsHaveEnoughFreeSpace', Source file 'CtrlVlmDfnModifyApiCallHandler.java', Line #671

Error message:
Error has been observed at the following site(s):
        |_ checkpoint ⇢ Modify volume definition
Stack trace:

Error context:
    Cannot grow the volume definition by 4091904KiB, as the following storage pool do not have enough free space:
    Node: sn1, StorPool: meta
    Node: sn2, StorPool: meta

Call backtrace:

    Method                                   Native Class:Line number
    ensureAllStorPoolsHaveEnoughFreeSpace    N      com.linbit.linstor.core.apicallhandler.controller.CtrlVlmDfnModifyApiCallHandler:671
    modifyVlmDfnInTransaction                N      com.linbit.linstor.core.apicallhandler.controller.CtrlVlmDfnModifyApiCallHandler:315
    lambda$modifyVlmDfn$0                    N      com.linbit.linstor.core.apicallhandler.controller.CtrlVlmDfnModifyApiCallHandler:179
    doInScope                                N      com.linbit.linstor.core.apicallhandler.ScopeRunner:149
    lambda$fluxInScope$0                     N      com.linbit.linstor.core.apicallhandler.ScopeRunner:76
    call                                     N      reactor.core.publisher.MonoCallable:91
    trySubscribeScalarMap                    N      reactor.core.publisher.FluxFlatMap:126
    subscribeOrReturn                        N      reactor.core.publisher.MonoFlatMapMany:49
    subscribe                                N      reactor.core.publisher.Flux:8343
    onNext                                   N      reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:188
    request                                  N      reactor.core.publisher.Operators$ScalarSubscription:2344
    onSubscribe                              N      reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:134
    subscribe                                N      reactor.core.publisher.MonoCurrentContext:35
    subscribe                                N      reactor.core.publisher.Flux:8357
    onNext                                   N      reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:188
    request                                  N      reactor.core.publisher.Operators$ScalarSubscription:2344
    onSubscribe                              N      reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:134
    subscribe                                N      reactor.core.publisher.MonoCurrentContext:35
    subscribe                                N      reactor.core.publisher.Mono:4252
    subscribeWith                            N      reactor.core.publisher.Mono:4363
    subscribe                                N      reactor.core.publisher.Mono:4223
    subscribe                                N      reactor.core.publisher.Mono:4159
    subscribe                                N      reactor.core.publisher.Mono:4131
    doFlux                                   N      com.linbit.linstor.api.rest.v1.RequestHelper:302
    modifyVolumeDefinition                   N      com.linbit.linstor.api.rest.v1.VolumeDefinitions:209
    invoke0                                  Y      jdk.internal.reflect.NativeMethodAccessorImpl:unknown
    invoke                                   N      jdk.internal.reflect.NativeMethodAccessorImpl:62
    invoke                                   N      jdk.internal.reflect.DelegatingMethodAccessorImpl:43
    invoke                                   N      java.lang.reflect.Method:566
    lambda$static$0                          N      org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory:52
    run                                      N      org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1:124
    invoke                                   N      org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher:167
    doDispatch                               N      org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker:159
    dispatch                                 N      org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher:79
    invoke                                   N      org.glassfish.jersey.server.model.ResourceMethodInvoker:469
    apply                                    N      org.glassfish.jersey.server.model.ResourceMethodInvoker:391
    apply                                    N      org.glassfish.jersey.server.model.ResourceMethodInvoker:80
    run                                      N      org.glassfish.jersey.server.ServerRuntime$1:253
    call                                     N      org.glassfish.jersey.internal.Errors$1:248
    call                                     N      org.glassfish.jersey.internal.Errors$1:244
    process                                  N      org.glassfish.jersey.internal.Errors:292
    process                                  N      org.glassfish.jersey.internal.Errors:274
    process                                  N      org.glassfish.jersey.internal.Errors:244
    runInScope                               N      org.glassfish.jersey.process.internal.RequestScope:265
    process                                  N      org.glassfish.jersey.server.ServerRuntime:232
    handle                                   N      org.glassfish.jersey.server.ApplicationHandler:680
    service                                  N      org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer:356
    run                                      N      org.glassfish.grizzly.http.server.HttpHandler$1:200
    doWork                                   N      org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker:569
    run                                      N      org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker:549
    run                                      N      java.lang.Thread:829


END OF ERROR REPORT.

I created a new volume. An error occurred expanding this volume to a size that exceeds the size of the metadata pool.

linstor resource-definition create volume1
linstor resource-definition set-property volume1 StorPoolName storage
linstor resource-definition set-property volume1 StorPoolNameDrbdMeta meta
linstor volume-definition create volume1 100MiB
linstor resource create volume1 --auto-place 2 --storage-pool storage
linstor volume-definition set-size volume1 0 4096MiB

$ linstor storage-pool list -p
+----------------------------------------------------------------------------------------------------------------------+
| StoragePool          | Node | Driver   | PoolName | FreeCapacity | TotalCapacity | CanSnapshots | State | SharedName |
|======================================================================================================================|
| DfltDisklessStorPool | sn1  | DISKLESS |          |              |               | False        | Ok    |            |
| DfltDisklessStorPool | sn2  | DISKLESS |          |              |               | False        | Ok    |            |
| meta                 | sn1  | ZFS      | pool1    |     1.74 GiB |      1.88 GiB | True         | Ok    |            |
| meta                 | sn2  | ZFS      | pool1    |     1.75 GiB |      1.88 GiB | True         | Ok    |            |
| storage              | sn1  | ZFS      | pool0    |    18.47 GiB |        59 GiB | True         | Ok    |            |
| storage              | sn2  | ZFS      | pool0    |    18.87 GiB |        59 GiB | True         | Ok    |            |
+----------------------------------------------------------------------------------------------------------------------+

Storage pool hava enogh space for volume size (4GiB < 18.47Gib), but failed by metadata pool capacity check (4GiB > 1.75GiB).

sos_2023-06-14_06-33-41.tar.gz
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix size-check for meta-pools with resize volume case kanno2inf/linstor-server
1 participant
@kanno2inf