Public dataset and model distribution based on DVC

[iesahin]

There are various licenses for downloadable datasets. dvc get and dvc import can check if there is a LICENSE file within a tracked directory and print this license, and ask the user for confirmation before download. This allows us to conform with attribution and copyright requirements in licenses like MIT or Apache.

For a Git repository directory in the form

.
├── README.md
├── fashion-mnist
│   ├── LICENSE
│   ├── raw
│   │   ├── t10k-images-idx3-ubyte.gz
│   │   ├── t10k-labels-idx1-ubyte.gz
│   │   ├── train-images-idx3-ubyte.gz
│   │   └── train-labels-idx1-ubyte.gz
│   └── raw.dvc

we use dvc get https://github.com/iterative/dataset-registry/fashion-mnist/raw.dvc to get the dataset.

At this point, instead of directly downloading, DVC can check whether there is a LICENSE file in the directory fashion-mnist/ and present it to the user for confirmation. The same is applicable to dvc import.

I think this should be the default behavior and an option like --skip-license-confirmation is also needed for scripts.

This provides a basis to provide all public datasets with different license restrictions in a single dataset registry.

[pmrowla]

This sounds like something that is just on the user to determine rather than DVC?

Git doesn't prompt you to comply with a license on git clone or when you add an external git repo as a submodule. (Or when I just use wget or curl to download a file directly from someone's github/gitlab/etc repo)

[iesahin]

I think about a use case like Kaggle datasets, but for the command line. We can have something like pip search (dvc search) to search these and use with dvc import. Suppose we have hundreds of datasets in the data registry, each will have different copyright and license restrictions. Kaggle shows the license for each dataset and Tensorflow has this disclaimer that basically says we don't host or distribute these, and we'll remove the dataset from listing if the owner asks for it. We'll be hosting and distributing the sets.

When we git clone from a public repository, we usually get a LICENSE or README file as well. For DVC, we get the individual files and this is similar to wget, but wget is just a tool that can run with any URL. DVC's use case is different, we have structured repositories that are published. And people can be very obsessive about these matters.

What I offer is a simple solution. If there is no LICENSE in the directory, simply use the current behavior. If there is one, ask for confirmation so that the legal responsibility of publishing these datasets can be met. I'm sure 99% of the users won't read that document but one person is enough to make a legal claim.

[pmrowla]

DVC doesn't host or distribute anything though, it's just tooling. I guess the line is blurred a bit when it comes to Studio, but it still seems to me like anything on the licensing/attribution side of things would be a Studio issue, and not a core DVC issue (similar to the difference between github/gitlab and git).

[iesahin]

Maintaining the legal requirements is certainly not DVC's responsibility, but providing a means for this may increase the adoption IMO. We can have a datasets.dvc.org domain that works like PyPI and users can obtain the sets without leaving from the command line. That needs a bit more metadata for the datasets perhaps, but we can create something similar to TF datasets rather quickly. It doesn't require much on top of the current dataset-registry.

[jorgeorpinel]

Agree that it could be a need, hypothetically. But I'd wait to see if any users or potential users express this first.

[pmrowla]

We can have a datasets.dvc.org domain that works like PyPI and users can obtain the sets without leaving from the command line.

This still sounds to me like a product that could potentially be built on top of (but separately from) DVC. Core DVC isn't a package manager and I don't think it should try to be one.

[iesahin]

DVC is not a package manager, but a dataset manager.

DVC has this functionality as shared remote. The one piece missing is associating metadata like description, license, etc. with .dvc files.

Agree that it could be a need, hypothetically. But I'd wait to see if any users or potential users express this first.

I doubt anybody will ask it first 😄 LICENSE files is necessary only when you index and distribute the datasets. I doubt anyone would need that. This is a feature request to increase the adoption and users don't need to increase the adoption I think :)

"We have a datasets collection in datasets.dvc.org and you can add yours with a PR, import any of them in your pipelines, use in any library or language, or simply get them to hack" would be a nice feature for anyone who works with data. It will increase the adoption extensively.

If another CLI tool will be used for this, it needs DVC's features.

[jorgeorpinel]

Many of DVC's features come from direct requests from the community. They tell us what DVC is missing for them to adopt it all the time. It's the beauty of having cooperation with your users 😃

[iesahin]

Yep, it's true. But think about PyPI: Does any typical Python developer need pip without PyPI? After building the central registry, they will probably want more features, whine about the search not working properly, etc. But before first providing them this feature, no one asks for this. I don't think our users build public dataset registries with DVC.

[dberenbaum]

@iesahin I think @pmrowla is suggesting that this could be built on top of DVC but be considered a separate product.

It seems like this issue is more of a feature request for a public dataset registry, with license confirmation being one of the requirements of that feature request. Would you agree @iesahin? Am I missing anything?

[iesahin]

We already have such a basic repository, though we don't (and maybe don't want to) advertise it. LICENSE prompt is a one step towards a public repository, and we can start to load more datasets to https://github.com/iterative/dataset-registry and start advertising it. A full-featured datasets.dvc.org may need more effort with additional changes but even distributing well-known datasets may bring us interest. :)

[dberenbaum]

I would suggest updating the title of the discussion to reflect that you are suggesting a dataset manager or public data registry, since this is probably a more interesting feature request. I'm not sure it makes sense for us to manage a giant registry of public data, but it's an interesting idea, and I can see the value of making dvc a popular means to access public data.

[shcheklein]

My 2cs on this. We had a few discussion around this with @dmpetrov and @efiop a few iterations back, and it is one of the topics where it's not that obvious how do draw the line. If generalize it beyond dataset licenses, we can think about dataset descriptions, semantic versions, tags (to do provenance), etc. It everything comes down usually to some attempts to include more meta-information about DVC tracked artifacts into DVC files (+ if needed some modifications to CLI to support it after, but this can be considered optional, since information can be consumed by other clients like Studio).

• what additional information do we include into files? If not in the files- how else can we preserve and version this information otherwise? E.g. we already included descriptions, we already have some basic stats like files sizes, number of files, etc. Pros for this approach- it's easier to manage (vs creating one more file, or a database, etc).
• on the other hand, we had an attempt like iterative/dvcx which also incorporates additional information about ML models into additional meta-files, and runs on top of DVC. But that product goes way beyond including additional info about an artifact, it has logic to act on that additional info.

To summarize. I would split the idea to save some additional info (like we did with descriptions) and idea to act on it (like we do with dvcx now). For the LICENSE we might have a field in the .dvc or in dvc.yaml (meta something?) that describes the license, but should dvc get act on it - I doubt to be honest. Can other tools (Studio? dvcx eventually, etc) benefit from this - yes.

[iesahin]

Including license document in a .dvc file may be too much. If it were just a string CC BY-NC-SA 4.0, that might work but thinking all the fancy stuff in this area, I prefer a separate document for licenses and descriptions. Maybe the pointer for these documents can be kept in .dvc files, as a path to the license, description, or readme files, but the content should better be kept separate, I think.

For versioning, DVC + Git already works. There may be different formats for the same dataset, e.g., having NumPy version of MNIST along with its binary IDX3 format, but these use cases are already covered. We can create different directories for each, mnist, mnist-np, mnist-matlab...

Along with LICENSE, there may be DESCRIPTION, and maybe a README. What other files a typical user needs? I don't think we need more versioning than already available, we can update the DESCRIPTION and commit to Git.

I'd prefer to add data discovery to dvc proper instead of building additional tools. Telling & selling, "here we have a new command for dvc, just type dvc datasets search cats to search for datasets with cat pictures" is easier than "we built a new tool that you can install and search for datasets and cat pictures (and do many more things.)" (dvcx looks like a nice tool on its own, BTW.)

If adding this prompt to dvc get and dvc import is too much, we can have a dvc datasets command with subcommands get, import, search, submit, etc. Instead of changing dvc gets behavior, dvc datasets get 'mnist/raw' shows the file in https://github.com/iterative/dataset-registry/mnist/LICENSE to the user for confirmation and runs dvc get https://github.com/iterative/dataset-registry/ mnist/raw.

[dmpetrov]

@iesahin it is a great idea but it feels like license information is a next-level abstraction - a packaging system (not Git as @pmrowla mentioned). We had a discussion a long time ago about making a packaging system on top of DVC. This looks like an essential feature for such kind of system, but not for dvc get.

[jorgeorpinel]

discussion a long time ago about making a packaging system on top of DVC

Are we referring to #2719 ? Data catalogs