Fix validation error when only Zarr assets are uploaded

[aaronkanzer]

Description of error

As documented in #1814, we receive the following validation error on the web application when uploading only Zarr asset(s) to a Dandiset (and not blobs):

assetsSummary: A Dandiset containing no files or zero bytes is not publishable

The error is raised from dandi-schema when a validation check occurs on the assetSummary field in the dandiset.yaml. See the models module:

@validator("assetsSummary")
def check_filesbytes(cls, values: AssetsSummary) -> AssetsSummary:
    if values.numberOfBytes == 0 or values.numberOfFiles == 0:
        raise ValueError(
            "A Dandiset containing no files or zero bytes is not publishable"
        )
    return values

And it arises because in the query below only the blob asset is evaluated:

dandi-archive/dandiapi/api/services/metadata/__init__.py

Line 127 in 0a4bf70

'numberOfBytes': 1 if version.assets.filter(blob__size__gt=0).exists() else 0,

Description of proposed fix

The proposed changes additionally evaluate the size of Zarr assets that are in the COMPLETE state to generate the assetSummary:numberOfBytes metadata field.

cc @kabilar

Closes #1814

[waxlamp]

I think we need a bit more explanation of the actual bug, and why this fixes it. We need to be careful with Zarr statuses because those have a complex relationship with Assets that results in (perhaps overly) complex semantics.

[aaronkanzer]

I think we need a bit more explanation of the actual bug, and why this fixes it. We need to be careful with Zarr statuses because those have a complex relationship with Assets that results in (perhaps overly) complex semantics.

In the case where there is no blob but just zarr FKs associated with the Asset, validation fails since the query of:

'numberOfBytes': 1 if version.assets.filter(blob__size__gt=0).exists() else 0,

only evaluates the blob Foreign Key. I stumbled upon this bug when uploading a dandiset of pure zarr if you'd like to replicate in DANDI Archive + dandischema's current state

[kabilar]

Hi @aaronkanzer, thank you for the original fix and pushing these changes upstream to DANDI.

Hi @waxlamp @jjnesbitt, I made some minor changes to the original comment above to further describe the issue and fix. Please take a look at it when you have a chance. Thank you.

[jjnesbitt]

The original implementation of this PR was modifying both the version_aggregate_assets_summary and validate_version_metadata functions, testing only version_aggregate_assets_summary. However, only validate_version_metadata required any change to address issue this PR was created for, and the changes to version_aggregate_assets_summary would have no impact. I've reworked this PR to reflect that.

@aaronkanzer I believe the test_validate_version_metadata_only_zarr_assets test reflects the situation in the PR description. If that's not the case, please elaborate.

[mvandenburgh]

LGTM

[aaronkanzer]

The original implementation of this PR was modifying both the version_aggregate_assets_summary and validate_version_metadata functions, testing only version_aggregate_assets_summary. However, only validate_version_metadata required any change to address issue this PR was created for, and the changes to version_aggregate_assets_summary would have no impact. I've reworked this PR to reflect that.

@aaronkanzer I believe the test_validate_version_metadata_only_zarr_assets test reflects the situation in the PR description. If that's not the case, please elaborate.

@jjnesbitt Just reviewed the tests you added -- thanks so much for this -- yes, this makes sense, and looks good to me

[dandibot]

🚀 PR was released in v0.3.113 🚀