New napari-plugin does not show up on the napari-hub

[haesleinhuepf]

Description

Hi all,

a new plugin napari-lattice does not show up on the napari-hub, even though the setup.cfg appears correct to us.

Can you give us guidance on how to publish this plugin on the hub?

Thanks!

CC @pr4deepr

See also

• setting up actions BioimageAnalysisCoreWEHI/napari_lattice#1

[haesleinhuepf]

Similar issue with this plugin :
https://pypi.org/project/napari-LF/

Cc @gschlafly

[richaagarwal]

@haesleinhuepf Thanks for opening this! There might have been a lag between when the plugin was released and when we retrieve new plugin data (though this should be happening every 5 minutes). Looks like those are showing up now.

Do you remember how much time had passed since release, at the time you weren't seeing the data on napari-hub?

[gschlafly]

@richaagarwal Our napari-LF plugin did not show up on napari-hub for at least four hours. When I checked 19 hours after the release, it was on napari-hub.

[richaagarwal]

Thank you for the info, @gschlafly!

It looks like there's a related issue #200 so I think it's worth more investigation at this point. I'll chat with the team to see when we might be able to prioritize this investigation.

[neuromusic]

here's another example: napari/napari#4862

napari-deepfinder

• released today on PyPI: https://pypi.org/project/napari-deepfinder/
• not found on the napari hub: https://www.napari-hub.org/plugins/napari-deepfinder

given that napari now uses the napari hub's API to get the list of available plugins, this is a problem that propagates to napari as well

[neuromusic]

oddly, for napari-deepfinder, the API is returning a partial response with npe2-related fields only (h/t @tlambert03):

https://api.napari-hub.org/plugins/napari-deepfinder

{
  "conda": [],
  "display_name": "",
  "plugin_types": [],
  "reader_file_extensions": [],
  "writer_file_extensions": [],
  "writer_save_layers": []
}

otoh, the expected response for a non-existant plugin should be an empty blob:

https://api.napari-hub.org/plugins/foobar

{}

[DragaDoncila]

Hmmm I thought the npe2 API was entirely switched off still? We shouldn't be seeing these fields at all, no? Did we only switch off the frontend @codemonkey800 ?

[codemonkey800]

@DragaDoncila hmm not sure, I thought we also switched it off on the backend but I'm not sure what was involved in doing that. maybe @klai95 @richaagarwal knows?

[DragaDoncila]

iirc the backend changes were just switching off the lambdas, I don't think we ever reverted the API changes. We should definitely gate it if we don't want to revert - not sure what the timelines are looking like for updating the backend work @richaagarwal

[richaagarwal]

@DragaDoncila That's correct, we never reverted the API changes. I don't know if there is a strong need to do this as we had discussed that no one else is really consuming the npe2 data yet, but maybe @neuromusic can weigh in here. But if needed, then I am in support of reverting all npe2-enabled features changes.

[DragaDoncila]

Do you think it's preferable to revert rather than just add a "gate" to hold the fields back? I think a lot of the API stuff won't change so it might be more effort to revert then add it back?

[codemonkey800]

Do you think it's preferable to revert rather than just add a "gate" to hold the fields back? I think a lot of the API stuff won't change so it might be more effort to revert then add it back?

@richaagarwal I've been looking into using a proper feature flag service on the hub frontend, I wonder if this is something we could use for the backend too in the future so things like this can be easier. They have a python SDK, so something we can use for the backend: https://help.split.io/hc/en-us/articles/360020359652-Python-SDK

[neuromusic]

why are we seeing these npe2 fields for napari-deepfinder (which is a real plugin but doesn't exist on the hub) and not for foobar (which isn't a real plugin)

[richaagarwal]

@klai95 @potating-potato any idea why we wouldn't be seeing data from https://pypi.org/pypi/napari-deepfinder/json populated yet on the hub side?

[DragaDoncila]

why are we seeing these npe2 fields for napari-deepfinder (which is a real plugin but doesn't exist on the hub) and not for foobar (which isn't a real plugin)

@neuromusic probably a quirk of when we're returning information. We decided early on that npe2 metadata would always be "available" in the metadata dictionary, so we return default values when we can't retrieve "real" information. When a plugin doesn't exist at all we do an early return of {}.

[neuromusic]

When a plugin doesn't exist at all

@DragaDoncila so even though it's not showing up, napari-deepfinder must exist somewhere in the hub backend?

[DragaDoncila]

@neuromusic correct - somewhere somehow, the napari hub backend is aware of napari-deepfinder and trying to process it, but not finding the PyPI metadata correctly.

[cnstt]

here's another example: napari/napari#4862

napari-deepfinder

* released today on PyPI: https://pypi.org/project/napari-deepfinder/

* not found on the napari hub: https://www.napari-hub.org/plugins/napari-deepfinder

The plugin is still not available on the napari-hub, do you know why? Because I just saw that more recent released plugins are available on the hub...

[liu-ziyang]

For some reason, the backend data for napari-deepfinder was corrupted, I deleted them and the repopulation of the data is then successful. The reason for the corruption is not yet known. I don't have any additional context but that the corrupted json contains nothing but "conda: []", this is not expected from the typical indexing, I believe this is probably related to npe2 somehow

[cnstt]

For some reason, the backend data for napari-deepfinder was corrupted, I deleted them and the repopulation of the data is then successful. The reason for the corruption is not yet known. I don't have any additional context but that the corrupted json contains nothing but "conda: []", this is not expected from the typical indexing, I believe this is probably related to npe2 somehow

Thanks for the fix! This is indeed somewhat strange xD

[richaagarwal]

@klai95 will be doing some exploration on this to understand what may be causing these intermittent lags reported here as well as in #200. In the meantime, if anyone notices this with any additional plugins, please let us know!

[pr4deepr]

Thanks for looking into this.
Just as a follow up from the original post, napari-lattice plugin appears on napari-hub but doesn't update when a new release is available on pypi.

v 0.1.5 in pypi

still v 0.1.4b in napari-hub

[liu-ziyang]

I digged around on the napari-lattice and I think I found the gap:
This may be related to https://github.com/chanzuckerberg/napari-hub/pull/605/files

when napari-lattice 0.1.5 published, we correctly fetched it and created a profile for it in our system, however when we populate the metadata, we used the non-versioned pypi API to get the latest, which seems to have not updated to correctly report the metadata for 0.1.5, as a result, the 0.1.5 profile is corrupted with 0.1.4b0 data when it was indexed. I will submit another PR to fix this

[tlambert03]

hey @potating-potato, let me know if there's anything I can do to facilitate here with npe2api. napari-lattice is looking good here: https://github.com/napari/npe2api/blob/main/public/manifest/napari-lattice.json

[liu-ziyang]

hey @potating-potato, let me know if there's anything I can do to facilitate here with npe2api. napari-lattice is looking good here: https://github.com/napari/npe2api/blob/main/public/manifest/napari-lattice.json

thanks for the help! I think I have found the issue and going to submit a PR to close this.

[richaagarwal]

Just to give a quick update here:

I think we may have pinpointed the cause here. Our hypothesis is that the lag may be due to PyPI's non-versioned endpoint sometimes taking time to catch up with the latest release data. For instance, https://pypi.org/pypi/mage-ai/json currently does not reflect latest release data available in https://pypi.org/pypi/mage-ai/0.2.1/json.

#608 outlines a potential fix to this.

[richaagarwal]

I am going to close this since we have #608 to track work on a potential fix. We can re-open this in the future if needed or the situation worsens.