Gh 621 bulk import #815

chb0github · 2023-10-02T03:41:14Z

Ok, this is working, but I am working on an improvement.

You see, if I fetch all the repo data, regardless of type, then I just remap it all. But, it turns out only docker local is our problem child, and the dockerApiVersion is the only thing I can't know with a single API call. I'll post the new code that carves these out as an exception for "performance" reasons - only those repos will get the double fetch which, definitely will result in a lot less calls, but for a lot more code (see below).

to use this tool, you run:

./bulkimport.sh --repos --users -g -h "${host}" > import.tf
# out="$RANDOM-out"
#terraform plan -generate-config-out generated.tf -out "${out} -parallelism=10
#terraform apply -parallelism=10

Duplicate resource requests are discarded

Remove repos project key default

next steps: handle the special case of docker local repos (we need to fetch the version number which requires an extra call) test management of netrc (we don't want to mess up someones file) exclude resources found in the local terraform state

Dockerfile

sample.tf

scripts/bulkimport.sh

scripts/repodata.jq

* master: Update devfile.yaml trying out spaces

…tainering this script isn't done, but it's also out-of-scope

alexhung · 2023-10-02T23:36:09Z

@chb0github Looks like I was wrong and Docker v1 is still very much supported.

chb0github · 2023-10-02T23:36:37Z

@alexhung - no, it doesn't actually try clicking v1 and hitting "submit"

alexhung · 2023-10-02T23:39:08Z

@chb0github No problem creating a local Docker v1 repository

chb0github · 2023-10-02T23:48:25Z

chb0github · 2023-10-04T16:38:53Z

So, do you want me to add the legacy code for v1, or is this good to commit and get feedback

alexhung · 2023-10-04T17:05:13Z

@chb0github So the Docker v1 issue seems to be a bug from Artifactory. The version you are using is 7.56.0, which is old. The local version I am running is 7.68.7 which support v1 again. Same with latest cloud version 7.69.4.

So we need to have Docker v1 support in the script.

…overed a bug in the netrc logic that needs fixing (removing for now).

…form-provider-artifactory into jfrogGH-621-bulk-import * 'jfrogGH-621-bulk-import' of github.com:chb0github/terraform-provider-artifactory: fix store_artifacts_locally docs to reflect reality JFrog Pipelines - Add Artifactory version to CHANGELOG.md Update CHANGELOG Update module path from 'v8' to 'v9' Add planmodifier.RequiresReplace() to "key" attribute Update CHANGELOG Migrate 'artifactory_backup' resource to terraform-plugin-framework Update CHANGELOG Update validation for "project_environments" to allow empty list Update documentation JFrog Pipelines - Add Artifactory version to CHANGELOG.md Update CHANGELOG Add "force_conan_authentication" attribute to local, virtual, and federated repo resources and data sources

chb0github · 2023-10-05T18:36:09Z

docker v1 support is added, and the script is updated

alexhung

Apart from the Terraform cli installation in Dockerfile, all looks fine to me.

sample.tf

Dockerfile

sample.tf

chb0github · 2023-10-09T17:21:14Z

I'm not particularly happy with the internal code formatting due to the need to output also with format to the import file, but using JQ was 6x faster than using the shell. I was shocked.

chb0github · 2023-10-09T17:21:57Z

If there is some input in the future, please tag me

scripts/bulkimport.sh

chb0github · 2023-10-09T20:16:55Z

@danielmkn waiting on you, buddy

danielmkn · 2023-10-09T20:40:33Z

@chb0github, can you please add some documentation on that?

chb0github · 2023-10-09T20:42:15Z

Where would you like me to put it? It was marked as no change log

Please confer with Alex. Also, the app is self documenting, per best practices

chb0github · 2023-10-13T16:49:13Z

@danielmkn - where would you like this documentation? And, did you consult with @alexhung ?

chb0github · 2023-10-16T21:44:27Z

Hey Guys, if we're not going to merge it in, then an explanation why would be great.

alexhung · 2023-10-16T23:04:39Z

Note We will put this PR on hold for the immediate future. Factors that contribute to this decision include:

This utility at its current state requires additional effort to support more resource types as well as 'productization', e.g. testing, documentation, etc.
JFrog's team inability to support this tool for the time being. Our plan for Q4 2023 is already set and we can revisit when we plan Q1 2024.

In the meantime, this PR will remain open and available for others to copy and use in their own trials and experiments. We welcome any suggestions or feedback so when we are ready to work on this in the future, we will have a better idea on what our customers want.

chb0github · 2023-12-05T22:01:43Z

Ok, So, having thought about this more, a more terraform idiosyncratic approach to this would be to :

Implement all artifactory resource types as data_sources
then, combine with a file data source and output the same format being generated by my script to a file_source

Although, right now the data source for repositories is still pending merge, and there aren't data sources for the other resources.

chb0github and others added 4 commits September 25, 2023 17:22

trying out spaces

9971e19

Update devfile.yaml

aed7081

Merge pull request jfrog#804 from jfrog/remove-repos-project-key-default

a5ecbd6

Remove repos project key default