diff --git a/.gitignore b/.gitignore index cdb0099d..dd736add 100644 --- a/.gitignore +++ b/.gitignore @@ -6,3 +6,9 @@ coverage/ .nyc_output *~ \#*# + +# dotenv environment variables file +.env +env.json + +*.env.list diff --git a/docs/add-new-harvester.md b/docs/add-new-harvester.md new file mode 100644 index 00000000..7fc083c7 --- /dev/null +++ b/docs/add-new-harvester.md @@ -0,0 +1,66 @@ +# Add a new harvest source + +This is a document in progress detailing the steps necessary to add a new harvest source. + +## Community Engagement +When adding a new harvest source, the points to be considered are located at https://docs.clearlydefined.io/adding-sources#adding-a-new-harvest-source. Please include these considerations and document them in your GitHub issue, similar to https://github.com/clearlydefined/service/issues/882. + +## Crawler +Need to implement: +- xxFetch, which is responsible for downloading package and registry information. +- xxExtract, which is responsible for creating a document based on the fetched information and queuing up scan tools and invoking source discovery. +- To enable the queuing of scan tools, config/map.js needs to be updated. + +Example commit: +``` +Commit: ea1618de0de4663b4d9aeecc9cdbc392edb2feba [ea1618d] +Parents: 968422b174 +Author: Nell Shamrell nells@microsoft.com +Date: October 27, 2021 4:16:41 PM +Committer: Nell Shamrell +adds support for fetching and extracting go packages to the crawler +``` + +## Service + +1. ClearlyDescribedSummarizer + +Example commit: +``` +Commit: 5e8b305f108b8cc9bd18c35ad5c626f71c081ef2 [5e8b305] +Parents: 6906969865 +Author: Nell Shamrell nells@microsoft.com +Date: August 3, 2021 2:55:44 PM +Committer: Nell Shamrell +Commit Date: October 27, 2021 3:56:37 PM +adds in code and test for determining urls for a go package +``` +2. /origin endpoint (for ui query) + +Example commit: +``` +Commit: 8c057670781451cfd7ca22337cecacae2124ac85 [8c05767] +Parents: 542c02763d +Author: Nell Shamrell nells@microsoft.com +Date: October 28, 2021 4:17:40 PM +Committer: Nell Shamrell +adds ability to get go package revisions through the service API +``` +3. update validation schemas + +Example commit: +``` +Commit: 21e11c45b97c06170f436db498c534ff079443d3 [21e11c4] +Parents: 90c8414909 +Author: Nell Shamrell nells@microsoft.com +Date: July 29, 2021 3:17:35 PM +Committer: Nell Shamrell +Commit Date: October 27, 2021 3:55:33 PM +adds go as a type in schemas +``` +## Documentation +Adaptation to reflect the new harvest source in the following documents: +- service/README.md +- service/swagger.yaml +- service/docs/determining-declared-license.md +- clearlydefined/docs/adding-sources.md