From b8822b9934e4f5591b3d094fd1a7719dc2217d2b Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva Date: Tue, 26 Mar 2024 12:24:40 -0400 Subject: [PATCH 01/61] Initial version --- doc/release-notes/6.2-release-notes.md | 242 +++++++++++++++++++++++++ 1 file changed, 242 insertions(+) create mode 100644 doc/release-notes/6.2-release-notes.md diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md new file mode 100644 index 00000000000..48903fb8b34 --- /dev/null +++ b/doc/release-notes/6.2-release-notes.md @@ -0,0 +1,242 @@ +# Dataverse 6.2 + +Please note: To read these instructions in full, please go to https://github.com/IQSS/dataverse/releases/tag/v6.2 rather than the list of releases, which will cut them off. + +This release brings new features, enhancements, and bug fixes to the Dataverse software. +Thank you to all of the community members who contributed code, suggestions, bug reports, and other assistance across the project. + +## Release highlights + +### New API Endpoint for Clearing an Individual Dataset From Solr + +A new Index API endpoint has been added allowing an admin to clear an individual dataset from Solr. + +### Return to Author Now Requires a Reason + +The Popup for returning to author now requires a reason that will be sent by email to the author. + +Please note that you can still type a creative and meaningful comment such as "The author would like to modify his dataset", "Files are missing", "Nothing to report" or "A curation report with comments and suggestions/instructions will follow in another email" that suits your situation. + +### Support for Using Multiple PID Providers + +This release adds support for using multiple PID (DOI, Handle, PermalLink) providers, multiple PID provider accounts +(managing a given protocol, authority,separator, shoulder combination), assigning PID provider accounts to specific collections, +and supporting transferred PIDs (where a PID is managed by an account when it's authority, separator, and/or shoulder don't match +the combination where the account can mint new PIDs). It also adds the ability for additional provider services beyond the existing +DataCite, EZId, Handle, and PermaLink providers to be dynamically added as separate jar files. + +These changes require per-provider settings rather than the global PID settings previously supported. While backward compatibility +for installations using a single PID Provider account is provided, updating to use the new microprofile settings is highly recommended +and will be required in a future version. + +New microprofile settings (where * indicates a provider id indicating which provider the setting is for): + +dataverse.pid.providers +dataverse.pid.default-provider +dataverse.pid.*.type +dataverse.pid.*.label +dataverse.pid.*.authority +dataverse.pid.*.shoulder +dataverse.pid.*.identifier-generation-style +dataverse.pid.*.datafile-pid-format +dataverse.pid.*.managed-list +dataverse.pid.*.excluded-list +dataverse.pid.*.datacite.mds-api-url +dataverse.pid.*.datacite.rest-api-url +dataverse.pid.*.datacite.username +dataverse.pid.*.datacite.password +dataverse.pid.*.ezid.api-url +dataverse.pid.*.ezid.username +dataverse.pid.*.ezid.password +dataverse.pid.*.permalink.base-url +dataverse.pid.*.permalink.separator +dataverse.pid.*.handlenet.index +dataverse.pid.*.handlenet.independent-service +dataverse.pid.*.handlenet.auth-handle +dataverse.pid.*.handlenet.key.path +dataverse.pid.*.handlenet.key.passphrase +dataverse.spi.pidproviders.directory + +### Geospatial Metadata Block Fields for North and South Renamed + +The Geospatial metadata block fields for north and south were labeled incorrectly as ‘Longitudes,’ as reported on #5645. After updating to this version of Dataverse, users will need to update all the endpoints that used ‘northLongitude’ and ‘southLongitude’ to ‘northLatitude’ and ‘southLatitude,’ respectively. + + +TODO: Whoever puts the release notes together should make sure there is the standard note about updating the schema after upgrading. + +### Add .QPJ and .QMD Extensions to Shapefile Handling + +- Support for `.qpj` and `.qmd` files in shapefile uploads has been introduced, ensuring that these files are properly recognized and handled as part of geospatial datasets in Dataverse. + +### Ingested Tabular Data Files Can Be Stored Without the Variable Name Header + +Tabular Data Ingest can now save the generated archival files with the list of variable names added as the first tab-delimited line. As the most significant effect of this feature. + +Access API will be able to take advantage of Direct Download for tab. files saved with these headers on S3 - since they no longer have to be generated and added to the streamed content on the fly. + +This behavior is controlled by the new setting `:StoreIngestedTabularFilesWithVarHeaders`. It is false by default, preserving the legacy behavior. When enabled, Dataverse will be able to handle both the newly ingested files, and any already-existing legacy files stored without these headers transparently to the user. E.g. the access API will continue delivering tab-delimited files **with** this header line, whether it needs to add it dynamically for the legacy files, or reading complete files directly from storage for the ones stored with it. + +An API for converting existing legacy tabular files will be added separately. [this line will need to be changed if we have time to add said API before 6.2 is released]. [TODO] + +### Search by License + +A new search facet called "License" has been added and will be displayed as long as there is more than one license in datasets and datafiles in browse/search results. This facet allow you to filter by license such as CC0, etc. + +Also, the Search API now handles license filtering using the `fq` parameter, for example : `/api/search?q=*&fq=license%3A%22CC0+1.0%22` for CC0 1.0. See PR #10204 + +### OAI-PMH Error Handling Has Been Improved + +OAI-PMH error handling has been improved to display a machine-readable error in XML rather than a 500 error with no further information. + +- /oai?foo=bar will show "No argument 'verb' found" +- /oai?verb=foo&verb=bar will show "Verb must be singular, given: '[foo, bar]'" + +### Rate Limiting Using JCache (With Hazelcast As Provided by Payara) + +The option to rate limit has been added to prevent users from over taxing the system either deliberately or by runaway automated processes. +Rate limiting can be configured on a tier level with tier 0 being reserved for guest users and tiers 1-any for authenticated users. +Superuser accounts are exempt from rate limiting. +Rate limits can be imposed on command APIs by configuring the tier, the command, and the hourly limit in the database. +Two database settings configure the rate limiting. +Note: If either of these settings exist in the database rate limiting will be enabled. +If neither setting exists rate limiting is disabled. + +`:RateLimitingDefaultCapacityTiers` is a comma separated list of default values for each tier. +In the following example, the default for tier `0` (guest users) is set to 10,000 calls per command per hour and tier `1` (authenticated users) is set to 20,000 calls per command per hour. +Tiers not specified in this setting will default to `-1` (No Limit). I.e., -d "10000" is equivalent to -d "10000,-1,-1,..." +`curl http://localhost:8080/api/admin/settings/:RateLimitingDefaultCapacityTiers -X PUT -d '10000,20000'` + +`:RateLimitingCapacityByTierAndAction` is a JSON object specifying the rate by tier and a list of actions (commands). +This allows for more control over the rate limit of individual API command calls. +In the following example, calls made by a guest user (tier 0) for API `GetLatestPublishedDatasetVersionCommand` is further limited to only 10 calls per hour, while an authenticated user (tier 1) will be able to make 30 calls per hour to the same API. +`curl http://localhost:8080/api/admin/settings/:RateLimitingCapacityByTierAndAction -X PUT -d '[{"tier": 0, "limitPerHour": 10, "actions": ["GetLatestPublishedDatasetVersionCommand", "GetPrivateUrlCommand", "GetDatasetCommand", "GetLatestAccessibleDatasetVersionCommand"]}, {"tier": 0, "limitPerHour": 1, "actions": ["CreateGuestbookResponseCommand", "UpdateDatasetVersionCommand", "DestroyDatasetCommand", "DeleteDataFileCommand", "FinalizeDatasetPublicationCommand", "PublishDatasetCommand"]}, {"tier": 1, "limitPerHour": 30, "actions": ["CreateGuestbookResponseCommand", "GetLatestPublishedDatasetVersionCommand", "GetPrivateUrlCommand", "GetDatasetCommand", "GetLatestAccessibleDatasetVersionCommand", "UpdateDatasetVersionCommand", "DestroyDatasetCommand", "DeleteDataFileCommand", "FinalizeDatasetPublicationCommand", "PublishDatasetCommand"]}]'` + +Hazelcast is configured in Payara and should not need any changes for this feature + +### Container Guide, Documentation for Faster Redeploy + +In the Container Guide, documentation for developers on how to quickly redeploy code has been added for Netbeans and improved for IntelliJ. + +Also in the context of containers, a new option to skip deployment has been added and the war file is now consistently named "dataverse.war" rather than having a version in the filename, such as "dataverse-6.1.war". This predictability makes tooling easier. + +Finally, an option to create tabs in the guides using [Sphinx Tabs](https://sphinx-tabs.readthedocs.io) has been added. (You can see the tabs in action in the "dev usage" page of the Container Guide.) To continue building the guides, you will need to install this new dependency by re-running `pip install -r requirements.txt`. + +### Universe Field in Variablemetadata Table Changed + +Universe field in variablemetadata table was changed from varchar(255) to text. The change was made to support longer strings in "universe" metadata field, similar to the rest of text fields in variablemetadata table. + +### Postgres Versions + +This release adds install script support for the new permissions model in Postgres versions 15+, and bumps FlyWay to support Postgres 16. + +Postgres 13 remains the version used with automated testing. + +### Listing Collection/Dataverse API + +Listing collection/dataverse role assignments via API still requires ManageDataversePermissions, but listing dataset role assignments via API now requires only ManageDatasetPermissions. + +### Missing Database Constraints + +This release adds two missing database constraints that will assure that the externalvocabularyvalue table only has one entry for each uri and that the oaiset table only has one set for each spec. (In the very unlikely case that your existing database has duplicate entries now, install would fail. This can be checked by running + +SELECT uri, count(*) FROM externalvocabularyvaluet group by uri; + +and + +SELECT spec, count(*) FROM oaiset group by spec; + +and then removing any duplicate rows (where count>1). + +TODO: Whoever puts the release notes together should make sure there is the standard note about reloading metadata blocks for the citation, astrophysics, and biomedical blocks (plus any others from other PRs) after upgrading. + +### Harvesting Client API + +The API endpoint `api/harvest/clients/{harvestingClientNickname}` has been extended to include the following fields: + +- `allowHarvestingMissingCVV`: enable/disable allowing datasets to be harvested with Controlled Vocabulary Values that existed in the originating Dataverse Project but are not in the harvesting Dataverse Project. Default is false. +Note: This setting is only available to the API and not currently accessible/settable via the UI + +### New QA Guide + +A new QA Guide is intended mostly for the core development team but may be of interest to contributors. + +### New Accounts Metrics API + +Users can retrieve new types of metrics related to user accounts. The new capabilities are [described](https://guides.dataverse.org/en/6.2/api/metrics.html) in the guides. + +### New canDownloadAtLeastOneFile API + +The GET canDownloadAtLeastOneFile (/api/datasets/{id}/versions/{versionId}/canDownloadAtLeastOneFile) endpoint has been created. + +This API endpoint indicates if the calling user can download at least one file from a dataset version. Note that Shibboleth group permissions are not considered. + +### Extended getVersionFiles API + +The response for getVersionFiles (/api/datasets/{id}/versions/{versionId}/files) endpoint has been modified to include a total count of records available (totalCount:x). +This will aid in pagination by allowing the caller to know how many pages can be iterated through. The existing API (getVersionFileCounts) to return the count will still be available. + +### Extended Metadata Blocks API + +The API endpoint `/api/metadatablocks/{block_id}` has been extended to include the following fields: + +- `isRequired`: Whether or not this field is required +- `displayOrder`: The display order of the field in create/edit forms +- `typeClass`: The type class of this field ("controlledVocabulary", "compound", or "primitive") + +### Evaluation Version Tutorial on the Containers Guide + +The Container Guide now containers a tutorial for running Dataverse in containers for demo or evaluation purposes: https://guides.dataverse.org/en/6.2/container + +### Get File Citation As JSON + +It is now possible to retrieve via API the file citation as it appears on the file landing page. It is formatted in HTML and encoded in JSON. + +This API is not for downloading various citation formats such as EndNote XML, RIS, or BibTeX. This functionality has been requested in https://github.com/IQSS/dataverse/issues/3140 and https://github.com/IQSS/dataverse/issues/9994 + +### Extended Files API + +The API endpoint `api/files/{id}` has been extended to support the following optional query parameters: + +- `includeDeaccessioned`: Indicates whether or not to consider deaccessioned dataset versions in the latest file search. (Default: `false`). +- `returnDatasetVersion`: Indicates whether or not to include the dataset version of the file in the response. (Default: `false`). + +A new endpoint `api/files/{id}/versions/{datasetVersionId}` has been created. This endpoint returns the file metadata present in the requested dataset version. To specify the dataset version, you can use ``:latest-published``, or ``:latest``, or ``:draft`` or ``1.0`` or any other available version identifier. + +The endpoint supports the `includeDeaccessioned` and `returnDatasetVersion` optional query parameters, as does the `api/files/{id}` endpoint. + +`api/files/{id}/draft` endpoint is no longer available in favor of the new endpoint `api/files/{id}/versions/{datasetVersionId}`, which can use the version identifier ``:draft`` (`api/files/{id}/versions/:draft`) to obtain the same result. + +### Endpoint Extended: Datasets, Dataverse Collections, and Datafiles + +The API endpoints for getting datasets, Dataverse collections, and datafiles have been extended to support the following optional 'returnOwners' query parameter. + +Including the parameter and setting it to true will add a hierarchy showing which dataset and dataverse collection(s) the object is part of to the json object returned. + +### Endpoint Fixed: Datasets Metadata + +The API endpoint `api/datasets/{id}/metadata` has been changed to default to the latest version of the dataset that the user has access. + +### Uningest/Reingest Options Available in the File Page Edit Menu + +New Uningest/Reingest options are available in the File Page Edit menu, allowing ingest errors to be cleared (by users who can published the associated dataset) +and (by superusers) for a successful ingest to be undone or retried (e.g. after a Dataverse version update or if ingest size limits are changed). +The /api/files//uningest api also now allows users who can publish the dataset to undo an ingest failure. + +### Publication Status Facet Restored + +In version 6.1, the publication status facet location was unintentionally moved to the bottom. In this version, we have restored the original order. + +### Permissions Required To Assign a Role Have Been Fixed + +The permissions required to assign a role have been fixed. It is no longer possible to assign a role that includes permissions that the assigning user doesn't have. + +### Binder Redirect + +If your installation is configured to use Binder, you should remove the old "girder_ythub" tool and replace it with the tool described at https://github.com/IQSS/dataverse-binder-redirect + +For more information, see #10360. + + +### Optional Croissant Exporter Support + +When a Dataverse installation is configured to use a metadata exporter for the [Croissant](https://github.com/mlcommons/croissant) format, the content of the JSON-LD in the `` of dataset landing pages will be replaced with that format. However, both JSON-LD and Croissant will still be available for download from the dataset page and API. From 4ef47742ed9462d80ea13945186434a02567bdc2 Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva Date: Tue, 26 Mar 2024 12:37:30 -0400 Subject: [PATCH 02/61] Delete files --- doc/release-notes/10101-qa-guide.md | 1 - ...17-api-metrics-add-user-accounts-number.md | 3 -- ...datasets-can-download-at-least-one-file.md | 3 -- ...onFiles-api-to-include-total-file-count.md | 2 - doc/release-notes/10216-metadatablocks.md | 5 --- doc/release-notes/10238-container-demo.md | 1 - doc/release-notes/10240-file-citation.md | 5 --- .../10280-get-file-api-extension.md | 10 ----- .../10286-return-owner-added-to-get-apis.md | 5 --- doc/release-notes/10297-metadata-api-fix.md | 1 - .../10318-uningest-and-reingest.md | 3 -- ...38-expose-and-sort-publish-status-facet.md | 1 - ...sign-roles-without-privilege-escalation.md | 1 - doc/release-notes/10360-binder-redirect.md | 3 -- .../10382-optional-croissant-exporter.md | 1 - doc/release-notes/3437-new-index-api-added.md | 4 -- doc/release-notes/3623-multipid.md | 37 ------------------- doc/release-notes/3702-return-to-author.md | 4 -- .../5645-geospatial-props-nslong-fix.md | 4 -- .../8134-add-qpj-qmd-extensions.md | 3 -- ...4-storing-tabular-files-with-varheaders.md | 6 --- ...482-make-licenses-searchable-faceatable.md | 6 --- .../9275-harvest-invalid-query-params.md | 4 -- doc/release-notes/9356-rate-limiting.md | 20 ---------- doc/release-notes/9590-faster-redeploy.md | 5 --- .../9728-universe-variablemetadata.md | 1 - doc/release-notes/9920-postgres16.md | 3 -- .../9926-list-role-assignments-permissions.md | 1 - doc/release-notes/9983-unique-constraints.md | 14 ------- ...harvest-metadata-values-not-in-cvv-list.md | 4 -- 30 files changed, 161 deletions(-) delete mode 100644 doc/release-notes/10101-qa-guide.md delete mode 100644 doc/release-notes/10117-api-metrics-add-user-accounts-number.md delete mode 100644 doc/release-notes/10155-datasets-can-download-at-least-one-file.md delete mode 100644 doc/release-notes/10202-extend-getVersionFiles-api-to-include-total-file-count.md delete mode 100644 doc/release-notes/10216-metadatablocks.md delete mode 100644 doc/release-notes/10238-container-demo.md delete mode 100644 doc/release-notes/10240-file-citation.md delete mode 100644 doc/release-notes/10280-get-file-api-extension.md delete mode 100644 doc/release-notes/10286-return-owner-added-to-get-apis.md delete mode 100644 doc/release-notes/10297-metadata-api-fix.md delete mode 100644 doc/release-notes/10318-uningest-and-reingest.md delete mode 100644 doc/release-notes/10338-expose-and-sort-publish-status-facet.md delete mode 100644 doc/release-notes/10342-assign-roles-without-privilege-escalation.md delete mode 100644 doc/release-notes/10360-binder-redirect.md delete mode 100644 doc/release-notes/10382-optional-croissant-exporter.md delete mode 100644 doc/release-notes/3437-new-index-api-added.md delete mode 100644 doc/release-notes/3623-multipid.md delete mode 100644 doc/release-notes/3702-return-to-author.md delete mode 100644 doc/release-notes/5645-geospatial-props-nslong-fix.md delete mode 100644 doc/release-notes/8134-add-qpj-qmd-extensions.md delete mode 100644 doc/release-notes/8524-storing-tabular-files-with-varheaders.md delete mode 100644 doc/release-notes/9060-7482-make-licenses-searchable-faceatable.md delete mode 100644 doc/release-notes/9275-harvest-invalid-query-params.md delete mode 100644 doc/release-notes/9356-rate-limiting.md delete mode 100644 doc/release-notes/9590-faster-redeploy.md delete mode 100644 doc/release-notes/9728-universe-variablemetadata.md delete mode 100644 doc/release-notes/9920-postgres16.md delete mode 100644 doc/release-notes/9926-list-role-assignments-permissions.md delete mode 100644 doc/release-notes/9983-unique-constraints.md delete mode 100644 doc/release-notes/9992-harvest-metadata-values-not-in-cvv-list.md diff --git a/doc/release-notes/10101-qa-guide.md b/doc/release-notes/10101-qa-guide.md deleted file mode 100644 index 11fbd7df2c4..00000000000 --- a/doc/release-notes/10101-qa-guide.md +++ /dev/null @@ -1 +0,0 @@ -A new QA Guide is intended mostly for the core development team but may be of interest to contributors. diff --git a/doc/release-notes/10117-api-metrics-add-user-accounts-number.md b/doc/release-notes/10117-api-metrics-add-user-accounts-number.md deleted file mode 100644 index 566815d6e5e..00000000000 --- a/doc/release-notes/10117-api-metrics-add-user-accounts-number.md +++ /dev/null @@ -1,3 +0,0 @@ -### New Accounts Metrics API - -Users can retrieve new types of metrics related to user accounts. The new capabilities are [described](https://guides.dataverse.org/en/6.2/api/metrics.html) in the guides. \ No newline at end of file diff --git a/doc/release-notes/10155-datasets-can-download-at-least-one-file.md b/doc/release-notes/10155-datasets-can-download-at-least-one-file.md deleted file mode 100644 index a0b0d02310a..00000000000 --- a/doc/release-notes/10155-datasets-can-download-at-least-one-file.md +++ /dev/null @@ -1,3 +0,0 @@ -The getCanDownloadAtLeastOneFile (/api/datasets/{id}/versions/{versionId}/canDownloadAtLeastOneFile) endpoint has been created. - -This API endpoint indicates if the calling user can download at least one file from a dataset version. Note that Shibboleth group permissions are not considered. diff --git a/doc/release-notes/10202-extend-getVersionFiles-api-to-include-total-file-count.md b/doc/release-notes/10202-extend-getVersionFiles-api-to-include-total-file-count.md deleted file mode 100644 index 80a71e9bb7e..00000000000 --- a/doc/release-notes/10202-extend-getVersionFiles-api-to-include-total-file-count.md +++ /dev/null @@ -1,2 +0,0 @@ -The response for getVersionFiles (/api/datasets/{id}/versions/{versionId}/files) endpoint has been modified to include a total count of records available (totalCount:x). -This will aid in pagination by allowing the caller to know how many pages can be iterated through. The existing API (getVersionFileCounts) to return the count will still be available. \ No newline at end of file diff --git a/doc/release-notes/10216-metadatablocks.md b/doc/release-notes/10216-metadatablocks.md deleted file mode 100644 index 59d9c1640a5..00000000000 --- a/doc/release-notes/10216-metadatablocks.md +++ /dev/null @@ -1,5 +0,0 @@ -The API endpoint `/api/metadatablocks/{block_id}` has been extended to include the following fields: - -- `isRequired`: Whether or not this field is required -- `displayOrder`: The display order of the field in create/edit forms -- `typeClass`: The type class of this field ("controlledVocabulary", "compound", or "primitive") diff --git a/doc/release-notes/10238-container-demo.md b/doc/release-notes/10238-container-demo.md deleted file mode 100644 index edc4db4b650..00000000000 --- a/doc/release-notes/10238-container-demo.md +++ /dev/null @@ -1 +0,0 @@ -The Container Guide now containers a tutorial for running Dataverse in containers for demo or evaluation purposes: https://guides.dataverse.org/en/6.2/container diff --git a/doc/release-notes/10240-file-citation.md b/doc/release-notes/10240-file-citation.md deleted file mode 100644 index fb747527669..00000000000 --- a/doc/release-notes/10240-file-citation.md +++ /dev/null @@ -1,5 +0,0 @@ -## Get file citation as JSON - -It is now possible to retrieve via API the file citation as it appears on the file landing page. It is formatted in HTML and encoded in JSON. - -This API is not for downloading various citation formats such as EndNote XML, RIS, or BibTeX. This functionality has been requested in https://github.com/IQSS/dataverse/issues/3140 and https://github.com/IQSS/dataverse/issues/9994 diff --git a/doc/release-notes/10280-get-file-api-extension.md b/doc/release-notes/10280-get-file-api-extension.md deleted file mode 100644 index 7ed70e93dc9..00000000000 --- a/doc/release-notes/10280-get-file-api-extension.md +++ /dev/null @@ -1,10 +0,0 @@ -The API endpoint `api/files/{id}` has been extended to support the following optional query parameters: - -- `includeDeaccessioned`: Indicates whether or not to consider deaccessioned dataset versions in the latest file search. (Default: `false`). -- `returnDatasetVersion`: Indicates whether or not to include the dataset version of the file in the response. (Default: `false`). - -A new endpoint `api/files/{id}/versions/{datasetVersionId}` has been created. This endpoint returns the file metadata present in the requested dataset version. To specify the dataset version, you can use ``:latest-published``, or ``:latest``, or ``:draft`` or ``1.0`` or any other available version identifier. - -The endpoint supports the `includeDeaccessioned` and `returnDatasetVersion` optional query parameters, as does the `api/files/{id}` endpoint. - -`api/files/{id}/draft` endpoint is no longer available in favor of the new endpoint `api/files/{id}/versions/{datasetVersionId}`, which can use the version identifier ``:draft`` (`api/files/{id}/versions/:draft`) to obtain the same result. diff --git a/doc/release-notes/10286-return-owner-added-to-get-apis.md b/doc/release-notes/10286-return-owner-added-to-get-apis.md deleted file mode 100644 index b0aba92f537..00000000000 --- a/doc/release-notes/10286-return-owner-added-to-get-apis.md +++ /dev/null @@ -1,5 +0,0 @@ -The API endpoints for getting datasets, Dataverse collections, and datafiles have been extended to support the following optional 'returnOwners' query parameter. - -Including the parameter and setting it to true will add a hierarchy showing which dataset and dataverse collection(s) the object is part of to the json object returned. - - diff --git a/doc/release-notes/10297-metadata-api-fix.md b/doc/release-notes/10297-metadata-api-fix.md deleted file mode 100644 index 11ee086af04..00000000000 --- a/doc/release-notes/10297-metadata-api-fix.md +++ /dev/null @@ -1 +0,0 @@ -The API endpoint `api/datasets/{id}/metadata` has been changed to default to the latest version of the dataset that the user has access. diff --git a/doc/release-notes/10318-uningest-and-reingest.md b/doc/release-notes/10318-uningest-and-reingest.md deleted file mode 100644 index 80ca6be57ea..00000000000 --- a/doc/release-notes/10318-uningest-and-reingest.md +++ /dev/null @@ -1,3 +0,0 @@ -New Uningest/Reingest options are available in the File Page Edit menu, allowing ingest errors to be cleared (by users who can published the associated dataset) -and (by superusers) for a successful ingest to be undone or retried (e.g. after a Dataverse version update or if ingest size limits are changed). -The /api/files//uningest api also now allows users who can publish the dataset to undo an ingest failure. diff --git a/doc/release-notes/10338-expose-and-sort-publish-status-facet.md b/doc/release-notes/10338-expose-and-sort-publish-status-facet.md deleted file mode 100644 index b2362ddb2c5..00000000000 --- a/doc/release-notes/10338-expose-and-sort-publish-status-facet.md +++ /dev/null @@ -1 +0,0 @@ -In version 6.1, the publication status facet location was unintentionally moved to the bottom. In this version, we have restored the original order. diff --git a/doc/release-notes/10342-assign-roles-without-privilege-escalation.md b/doc/release-notes/10342-assign-roles-without-privilege-escalation.md deleted file mode 100644 index a4ef743f50d..00000000000 --- a/doc/release-notes/10342-assign-roles-without-privilege-escalation.md +++ /dev/null @@ -1 +0,0 @@ -The permissions required to assign a role have been fixed. It is no longer possible to assign a role that includes permissions that the assigning user doesn't have. \ No newline at end of file diff --git a/doc/release-notes/10360-binder-redirect.md b/doc/release-notes/10360-binder-redirect.md deleted file mode 100644 index fcf5feea69e..00000000000 --- a/doc/release-notes/10360-binder-redirect.md +++ /dev/null @@ -1,3 +0,0 @@ -If your installation is configured to use Binder, you should remove the old "girder_ythub" tool and replace it with the tool described at https://github.com/IQSS/dataverse-binder-redirect - -For more information, see #10360. diff --git a/doc/release-notes/10382-optional-croissant-exporter.md b/doc/release-notes/10382-optional-croissant-exporter.md deleted file mode 100644 index e4c96115825..00000000000 --- a/doc/release-notes/10382-optional-croissant-exporter.md +++ /dev/null @@ -1 +0,0 @@ -When a Dataverse installation is configured to use a metadata exporter for the [Croissant](https://github.com/mlcommons/croissant) format, the content of the JSON-LD in the `` of dataset landing pages will be replaced with that format. However, both JSON-LD and Croissant will still be available for download from the dataset page and API. \ No newline at end of file diff --git a/doc/release-notes/3437-new-index-api-added.md b/doc/release-notes/3437-new-index-api-added.md deleted file mode 100644 index 2f40c65073f..00000000000 --- a/doc/release-notes/3437-new-index-api-added.md +++ /dev/null @@ -1,4 +0,0 @@ -(this API was added as a side feature of the pr #10222. the main point of the pr was an improvement in the OAI set housekeeping logic, I believe it's too obscure part of the system to warrant a relase note by itself. but the new API below needs to be announced). - -A new Index API endpoint has been added allowing an admin to clear an individual dataset from Solr. - diff --git a/doc/release-notes/3623-multipid.md b/doc/release-notes/3623-multipid.md deleted file mode 100644 index 8c13eb1aec6..00000000000 --- a/doc/release-notes/3623-multipid.md +++ /dev/null @@ -1,37 +0,0 @@ -This release adds support for using multiple PID (DOI, Handle, PermalLink) providers, multiple PID provider accounts -(managing a given protocol, authority,separator, shoulder combination), assigning PID provider accounts to specific collections, -and supporting transferred PIDs (where a PID is managed by an account when it's authority, separator, and/or shoulder don't match -the combination where the account can mint new PIDs). It also adds the ability for additional provider services beyond the existing -DataCite, EZId, Handle, and PermaLink providers to be dynamically added as separate jar files. - -These changes require per-provider settings rather than the global PID settings previously supported. While backward compatibility -for installations using a single PID Provider account is provided, updating to use the new microprofile settings is highly recommended -and will be required in a future version. - -New microprofile settings (where * indicates a provider id indicating which provider the setting is for): - -dataverse.pid.providers -dataverse.pid.default-provider -dataverse.pid.*.type -dataverse.pid.*.label -dataverse.pid.*.authority -dataverse.pid.*.shoulder -dataverse.pid.*.identifier-generation-style -dataverse.pid.*.datafile-pid-format -dataverse.pid.*.managed-list -dataverse.pid.*.excluded-list -dataverse.pid.*.datacite.mds-api-url -dataverse.pid.*.datacite.rest-api-url -dataverse.pid.*.datacite.username -dataverse.pid.*.datacite.password -dataverse.pid.*.ezid.api-url -dataverse.pid.*.ezid.username -dataverse.pid.*.ezid.password -dataverse.pid.*.permalink.base-url -dataverse.pid.*.permalink.separator -dataverse.pid.*.handlenet.index -dataverse.pid.*.handlenet.independent-service -dataverse.pid.*.handlenet.auth-handle -dataverse.pid.*.handlenet.key.path -dataverse.pid.*.handlenet.key.passphrase -dataverse.spi.pidproviders.directory diff --git a/doc/release-notes/3702-return-to-author.md b/doc/release-notes/3702-return-to-author.md deleted file mode 100644 index aa7dd9feaef..00000000000 --- a/doc/release-notes/3702-return-to-author.md +++ /dev/null @@ -1,4 +0,0 @@ -### Return to author - -Popup for returning to author now requires a reason that will be sent by email to the author. -Please note that you can still type a creative and meaningful comment such as "The author would like to modify his dataset", "Files are missing", "Nothing to report" or "A curation report with comments and suggestions/instructions will follow in another email" that suits your situation. \ No newline at end of file diff --git a/doc/release-notes/5645-geospatial-props-nslong-fix.md b/doc/release-notes/5645-geospatial-props-nslong-fix.md deleted file mode 100644 index 4004bf38c78..00000000000 --- a/doc/release-notes/5645-geospatial-props-nslong-fix.md +++ /dev/null @@ -1,4 +0,0 @@ -Across the application, the Geospatial metadata block fields for north and south were labeled incorrectly as ‘Longitudes,’ as reported on #5645. After updating to this version of Dataverse, users will need to update all the endpoints that used ‘northLongitude’ and ‘southLongitude’ to ‘northLatitude’ and ‘southLatitude,’ respectively. - - -TODO: Whoever puts the release notes together should make sure there is the standard note about updating the schema after upgrading. \ No newline at end of file diff --git a/doc/release-notes/8134-add-qpj-qmd-extensions.md b/doc/release-notes/8134-add-qpj-qmd-extensions.md deleted file mode 100644 index 65f4485354b..00000000000 --- a/doc/release-notes/8134-add-qpj-qmd-extensions.md +++ /dev/null @@ -1,3 +0,0 @@ -Add .qpj and .qmd Extensions to Shapefile Handling - -- Support for `.qpj` and `.qmd` files in shapefile uploads has been introduced, ensuring that these files are properly recognized and handled as part of geospatial datasets in Dataverse. diff --git a/doc/release-notes/8524-storing-tabular-files-with-varheaders.md b/doc/release-notes/8524-storing-tabular-files-with-varheaders.md deleted file mode 100644 index f7034c846f6..00000000000 --- a/doc/release-notes/8524-storing-tabular-files-with-varheaders.md +++ /dev/null @@ -1,6 +0,0 @@ -Tabular Data Ingest can now save the generated archival files with the list of variable names added as the first tab-delimited line. As the most significant effect of this feature, -Access API will be able to take advantage of Direct Download for tab. files saved with these headers on S3 - since they no longer have to be generated and added to the streamed content on the fly. - -This behavior is controlled by the new setting `:StoreIngestedTabularFilesWithVarHeaders`. It is false by default, preserving the legacy behavior. When enabled, Dataverse will be able to handle both the newly ingested files, and any already-existing legacy files stored without these headers transparently to the user. E.g. the access API will continue delivering tab-delimited files **with** this header line, whether it needs to add it dynamically for the legacy files, or reading complete files directly from storage for the ones stored with it. - -An API for converting existing legacy tabular files will be added separately. [this line will need to be changed if we have time to add said API before 6.2 is released]. \ No newline at end of file diff --git a/doc/release-notes/9060-7482-make-licenses-searchable-faceatable.md b/doc/release-notes/9060-7482-make-licenses-searchable-faceatable.md deleted file mode 100644 index 1758fd4de22..00000000000 --- a/doc/release-notes/9060-7482-make-licenses-searchable-faceatable.md +++ /dev/null @@ -1,6 +0,0 @@ -### Search by License - -A browse/search facet called "License" has been added and will be displayed as long as there is more than one license in datasets and datafiles in browse/search results. This facet allow you to filter by license such as CC0, etc. -Also, the Search API now handles license filtering using the `fq` parameter, for example : `/api/search?q=*&fq=license%3A%22CC0+1.0%22` for CC0 1.0. See PR #10204 - - diff --git a/doc/release-notes/9275-harvest-invalid-query-params.md b/doc/release-notes/9275-harvest-invalid-query-params.md deleted file mode 100644 index 33d7c7bac13..00000000000 --- a/doc/release-notes/9275-harvest-invalid-query-params.md +++ /dev/null @@ -1,4 +0,0 @@ -OAI-PMH error handling has been improved to display a machine-readable error in XML rather than a 500 error with no further information. - -- /oai?foo=bar will show "No argument 'verb' found" -- /oai?verb=foo&verb=bar will show "Verb must be singular, given: '[foo, bar]'" diff --git a/doc/release-notes/9356-rate-limiting.md b/doc/release-notes/9356-rate-limiting.md deleted file mode 100644 index 1d68669af26..00000000000 --- a/doc/release-notes/9356-rate-limiting.md +++ /dev/null @@ -1,20 +0,0 @@ -## Rate Limiting using JCache (with Hazelcast as provided by Payara) -The option to rate limit has been added to prevent users from over taxing the system either deliberately or by runaway automated processes. -Rate limiting can be configured on a tier level with tier 0 being reserved for guest users and tiers 1-any for authenticated users. -Superuser accounts are exempt from rate limiting. -Rate limits can be imposed on command APIs by configuring the tier, the command, and the hourly limit in the database. -Two database settings configure the rate limiting. -Note: If either of these settings exist in the database rate limiting will be enabled. -If neither setting exists rate limiting is disabled. - -`:RateLimitingDefaultCapacityTiers` is a comma separated list of default values for each tier. -In the following example, the default for tier `0` (guest users) is set to 10,000 calls per command per hour and tier `1` (authenticated users) is set to 20,000 calls per command per hour. -Tiers not specified in this setting will default to `-1` (No Limit). I.e., -d "10000" is equivalent to -d "10000,-1,-1,..." -`curl http://localhost:8080/api/admin/settings/:RateLimitingDefaultCapacityTiers -X PUT -d '10000,20000'` - -`:RateLimitingCapacityByTierAndAction` is a JSON object specifying the rate by tier and a list of actions (commands). -This allows for more control over the rate limit of individual API command calls. -In the following example, calls made by a guest user (tier 0) for API `GetLatestPublishedDatasetVersionCommand` is further limited to only 10 calls per hour, while an authenticated user (tier 1) will be able to make 30 calls per hour to the same API. -`curl http://localhost:8080/api/admin/settings/:RateLimitingCapacityByTierAndAction -X PUT -d '[{"tier": 0, "limitPerHour": 10, "actions": ["GetLatestPublishedDatasetVersionCommand", "GetPrivateUrlCommand", "GetDatasetCommand", "GetLatestAccessibleDatasetVersionCommand"]}, {"tier": 0, "limitPerHour": 1, "actions": ["CreateGuestbookResponseCommand", "UpdateDatasetVersionCommand", "DestroyDatasetCommand", "DeleteDataFileCommand", "FinalizeDatasetPublicationCommand", "PublishDatasetCommand"]}, {"tier": 1, "limitPerHour": 30, "actions": ["CreateGuestbookResponseCommand", "GetLatestPublishedDatasetVersionCommand", "GetPrivateUrlCommand", "GetDatasetCommand", "GetLatestAccessibleDatasetVersionCommand", "UpdateDatasetVersionCommand", "DestroyDatasetCommand", "DeleteDataFileCommand", "FinalizeDatasetPublicationCommand", "PublishDatasetCommand"]}]'` - -Hazelcast is configured in Payara and should not need any changes for this feature \ No newline at end of file diff --git a/doc/release-notes/9590-faster-redeploy.md b/doc/release-notes/9590-faster-redeploy.md deleted file mode 100644 index ed903849444..00000000000 --- a/doc/release-notes/9590-faster-redeploy.md +++ /dev/null @@ -1,5 +0,0 @@ -In the Container Guide, documentation for developers on how to quickly redeploy code has been added for Netbeans and improved for IntelliJ. - -Also in the context of containers, a new option to skip deployment has been added and the war file is now consistently named "dataverse.war" rather than having a version in the filename, such as "dataverse-6.1.war". This predictability makes tooling easier. - -Finally, an option to create tabs in the guides using [Sphinx Tabs](https://sphinx-tabs.readthedocs.io) has been added. (You can see the tabs in action in the "dev usage" page of the Container Guide.) To continue building the guides, you will need to install this new dependency by re-running `pip install -r requirements.txt`. \ No newline at end of file diff --git a/doc/release-notes/9728-universe-variablemetadata.md b/doc/release-notes/9728-universe-variablemetadata.md deleted file mode 100644 index 66a2daf151b..00000000000 --- a/doc/release-notes/9728-universe-variablemetadata.md +++ /dev/null @@ -1 +0,0 @@ -universe field in variablemetadata table was changed from varchar(255) to text. The change was made to support longer strings in "universe" metadata field, similar to the rest of text fields in variablemetadata table. diff --git a/doc/release-notes/9920-postgres16.md b/doc/release-notes/9920-postgres16.md deleted file mode 100644 index 8aab76e98b9..00000000000 --- a/doc/release-notes/9920-postgres16.md +++ /dev/null @@ -1,3 +0,0 @@ -This release adds install script support for the new permissions model in Postgres versions 15+, and bumps FlyWay to support Postgres 16. - -Postgres 13 remains the version used with automated testing. diff --git a/doc/release-notes/9926-list-role-assignments-permissions.md b/doc/release-notes/9926-list-role-assignments-permissions.md deleted file mode 100644 index 43cd83dc5c9..00000000000 --- a/doc/release-notes/9926-list-role-assignments-permissions.md +++ /dev/null @@ -1 +0,0 @@ -Listing collction/dataverse role assignments via API still requires ManageDataversePermissions, but listing dataset role assignments via API now requires only ManageDatasetPermissions. diff --git a/doc/release-notes/9983-unique-constraints.md b/doc/release-notes/9983-unique-constraints.md deleted file mode 100644 index d889beb0718..00000000000 --- a/doc/release-notes/9983-unique-constraints.md +++ /dev/null @@ -1,14 +0,0 @@ -This release adds two missing database constraints that will assure that the externalvocabularyvalue table only has one entry for each uri and that the oaiset table only has one set for each spec. (In the very unlikely case that your existing database has duplicate entries now, install would fail. This can be checked by running - -SELECT uri, count(*) FROM externalvocabularyvaluet group by uri; - -and - -SELECT spec, count(*) FROM oaiset group by spec; - -and then removing any duplicate rows (where count>1). - - - - -TODO: Whoever puts the release notes together should make sure there is the standard note about reloading metadata blocks for the citation, astrophysics, and biomedical blocks (plus any others from other PRs) after upgrading. \ No newline at end of file diff --git a/doc/release-notes/9992-harvest-metadata-values-not-in-cvv-list.md b/doc/release-notes/9992-harvest-metadata-values-not-in-cvv-list.md deleted file mode 100644 index 88ca6cf0e79..00000000000 --- a/doc/release-notes/9992-harvest-metadata-values-not-in-cvv-list.md +++ /dev/null @@ -1,4 +0,0 @@ -The API endpoint `api/harvest/clients/{harvestingClientNickname}` has been extended to include the following fields: - -- `allowHarvestingMissingCVV`: enable/disable allowing datasets to be harvested with Controlled Vocabulary Values that existed in the originating Dataverse Project but are not in the harvesting Dataverse Project. Default is false. -Note: This setting is only available to the API and not currently accessible/settable via the UI \ No newline at end of file From 0400037b3c83b64a466b68e64b92caefa593ea74 Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva Date: Tue, 26 Mar 2024 14:08:47 -0400 Subject: [PATCH 03/61] Formating --- doc/release-notes/6.2-release-notes.md | 187 ++++++++++++++++--------- 1 file changed, 117 insertions(+), 70 deletions(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index 48903fb8b34..57f188bdd20 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -7,10 +7,6 @@ Thank you to all of the community members who contributed code, suggestions, bug ## Release highlights -### New API Endpoint for Clearing an Individual Dataset From Solr - -A new Index API endpoint has been added allowing an admin to clear an individual dataset from Solr. - ### Return to Author Now Requires a Reason The Popup for returning to author now requires a reason that will be sent by email to the author. @@ -31,38 +27,119 @@ and will be required in a future version. New microprofile settings (where * indicates a provider id indicating which provider the setting is for): -dataverse.pid.providers -dataverse.pid.default-provider -dataverse.pid.*.type -dataverse.pid.*.label -dataverse.pid.*.authority -dataverse.pid.*.shoulder -dataverse.pid.*.identifier-generation-style -dataverse.pid.*.datafile-pid-format -dataverse.pid.*.managed-list -dataverse.pid.*.excluded-list -dataverse.pid.*.datacite.mds-api-url -dataverse.pid.*.datacite.rest-api-url -dataverse.pid.*.datacite.username -dataverse.pid.*.datacite.password -dataverse.pid.*.ezid.api-url -dataverse.pid.*.ezid.username -dataverse.pid.*.ezid.password -dataverse.pid.*.permalink.base-url -dataverse.pid.*.permalink.separator -dataverse.pid.*.handlenet.index -dataverse.pid.*.handlenet.independent-service -dataverse.pid.*.handlenet.auth-handle -dataverse.pid.*.handlenet.key.path -dataverse.pid.*.handlenet.key.passphrase -dataverse.spi.pidproviders.directory +- `dataverse.pid.providers` +- `dataverse.pid.default-provider` +- `dataverse.pid.*.type` +- `dataverse.pid.*.label` +- `dataverse.pid.*.authority` +- `dataverse.pid.*.shoulder` +- `dataverse.pid.*.identifier-generation-style` +- `dataverse.pid.*.datafile-pid-format` +- `dataverse.pid.*.managed-list` +- `dataverse.pid.*.excluded-list` +- `dataverse.pid.*.datacite.mds-api-url` +- `dataverse.pid.*.datacite.rest-api-url` +- `dataverse.pid.*.datacite.username` +- `dataverse.pid.*.datacite.password` +- `dataverse.pid.*.ezid.api-url` +- `dataverse.pid.*.ezid.username` +- `dataverse.pid.*.ezid.password` +- `dataverse.pid.*.permalink.base-url` +- `dataverse.pid.*.permalink.separator` +- `dataverse.pid.*.handlenet.index` +- `dataverse.pid.*.handlenet.independent-service` +- `dataverse.pid.*.handlenet.auth-handle` +- `dataverse.pid.*.handlenet.key.path` +- `dataverse.pid.*.handlenet.key.passphrase` +- `dataverse.spi.pidproviders.directory` -### Geospatial Metadata Block Fields for North and South Renamed +### Rate Limiting Using JCache (With Hazelcast As Provided by Payara) -The Geospatial metadata block fields for north and south were labeled incorrectly as ‘Longitudes,’ as reported on #5645. After updating to this version of Dataverse, users will need to update all the endpoints that used ‘northLongitude’ and ‘southLongitude’ to ‘northLatitude’ and ‘southLatitude,’ respectively. +The option to rate limit has been added to prevent users from over taxing the system either deliberately or by runaway automated processes. +Rate limiting can be configured on a tier level with tier 0 being reserved for guest users and tiers 1-any for authenticated users. +Superuser accounts are exempt from rate limiting. +Rate limits can be imposed on command APIs by configuring the tier, the command, and the hourly limit in the database. +Two database settings configure the rate limiting. +Note: If either of these settings exist in the database rate limiting will be enabled. +If neither setting exists rate limiting is disabled. +`:RateLimitingDefaultCapacityTiers` is a comma separated list of default values for each tier. +In the following example, the default for tier `0` (guest users) is set to 10,000 calls per command per hour and tier `1` (authenticated users) is set to 20,000 calls per command per hour. +Tiers not specified in this setting will default to `-1` (No Limit). I.e., -d "10000" is equivalent to -d "10000,-1,-1,..." -TODO: Whoever puts the release notes together should make sure there is the standard note about updating the schema after upgrading. +`'curl http://localhost:8080/api/admin/settings/:RateLimitingDefaultCapacityTiers -X PUT -d '10000,20000'` + +`:RateLimitingCapacityByTierAndAction` is a JSON object specifying the rate by tier and a list of actions (commands). +This allows for more control over the rate limit of individual API command calls. +In the following example, calls made by a guest user (tier 0) for API `GetLatestPublishedDatasetVersionCommand` is further limited to only 10 calls per hour, while an authenticated user (tier 1) will be able to make 30 calls per hour to the same API. + +`curl http://localhost:8080/api/admin/settings/:RateLimitingCapacityByTierAndAction -X PUT -d '[{"tier": 0, "limitPerHour": 10, "actions": ["GetLatestPublishedDatasetVersionCommand", "GetPrivateUrlCommand", "GetDatasetCommand", "GetLatestAccessibleDatasetVersionCommand"]}, {"tier": 0, "limitPerHour": 1, "actions": ["CreateGuestbookResponseCommand", "UpdateDatasetVersionCommand", "DestroyDatasetCommand", "DeleteDataFileCommand", "FinalizeDatasetPublicationCommand", "PublishDatasetCommand"]}, {"tier": 1, "limitPerHour": 30, "actions": ["CreateGuestbookResponseCommand", "GetLatestPublishedDatasetVersionCommand", "GetPrivateUrlCommand", "GetDatasetCommand", "GetLatestAccessibleDatasetVersionCommand", "UpdateDatasetVersionCommand", "DestroyDatasetCommand", "DeleteDataFileCommand", "FinalizeDatasetPublicationCommand", "PublishDatasetCommand"]}]'` + + +``` +curl http://localhost:8080/api/admin/settings/:RateLimitingCapacityByTierAndAction -X PUT -d '[{ + "tier": 0, + "limitPerHour": 10, + "actions": [ + "GetLatestPublishedDatasetVersionCommand", + "GetPrivateUrlCommand", + "GetDatasetCommand", + "GetLatestAccessibleDatasetVersionCommand" + ] +}, +{ + "tier": 0, + "limitPerHour": 1, + "actions": [ + "CreateGuestbookResponseCommand", + "UpdateDatasetVersionCommand", + "DestroyDatasetCommand", + "DeleteDataFileCommand", + "FinalizeDatasetPublicationCommand", + "PublishDatasetCommand" + ] +}, +{ + "tier": 1, + "limitPerHour": 30, + "actions": [ + "CreateGuestbookResponseCommand", + "GetLatestPublishedDatasetVersionCommand", + "GetPrivateUrlCommand", + "GetDatasetCommand", + "GetLatestAccessibleDatasetVersionCommand", + "UpdateDatasetVersionCommand", + "DestroyDatasetCommand", + "DeleteDataFileCommand", + "FinalizeDatasetPublicationCommand", + "PublishDatasetCommand" + ] +}]' +``` + +### Binder Redirect + +If your installation is configured to use Binder, you should remove the old "girder_ythub" tool and replace it with the tool described at https://github.com/IQSS/dataverse-binder-redirect + +For more information, see [#10360](https://github.com/IQSS/dataverse/issues/10360). + +### Optional Croissant Exporter Support + +When a Dataverse installation is configured to use a metadata exporter for the [Croissant](https://github.com/mlcommons/croissant) format, the content of the JSON-LD in the `` of dataset landing pages will be replaced with that format. However, both JSON-LD and Croissant will still be available for download from the dataset page and API. + + + +Hazelcast is configured in Payara and should not need any changes for this feature + +### Search by License + +A new search facet called "License" has been added and will be displayed as long as there is more than one license in datasets and datafiles in browse/search results. This facet allow you to filter by license such as CC0, etc. + +Also, the Search API now handles license filtering using the `fq` parameter, for example : `/api/search?q=*&fq=license%3A%22CC0+1.0%22` for CC0 1.0. See PR #10204 + +### New API Endpoint for Clearing an Individual Dataset From Solr + +A new Index API endpoint has been added allowing an admin to clear an individual dataset from Solr. ### Add .QPJ and .QMD Extensions to Shapefile Handling @@ -78,11 +155,12 @@ This behavior is controlled by the new setting `:StoreIngestedTabularFilesWithVa An API for converting existing legacy tabular files will be added separately. [this line will need to be changed if we have time to add said API before 6.2 is released]. [TODO] -### Search by License +### Geospatial Metadata Block Fields for North and South Renamed -A new search facet called "License" has been added and will be displayed as long as there is more than one license in datasets and datafiles in browse/search results. This facet allow you to filter by license such as CC0, etc. +The Geospatial metadata block fields for north and south were labeled incorrectly as ‘Longitudes,’ as reported on #5645. After updating to this version of Dataverse, users will need to update all the endpoints that used ‘northLongitude’ and ‘southLongitude’ to ‘northLatitude’ and ‘southLatitude,’ respectively. -Also, the Search API now handles license filtering using the `fq` parameter, for example : `/api/search?q=*&fq=license%3A%22CC0+1.0%22` for CC0 1.0. See PR #10204 + +TODO: Whoever puts the release notes together should make sure there is the standard note about updating the schema after upgrading. ### OAI-PMH Error Handling Has Been Improved @@ -91,28 +169,6 @@ OAI-PMH error handling has been improved to display a machine-readable error in - /oai?foo=bar will show "No argument 'verb' found" - /oai?verb=foo&verb=bar will show "Verb must be singular, given: '[foo, bar]'" -### Rate Limiting Using JCache (With Hazelcast As Provided by Payara) - -The option to rate limit has been added to prevent users from over taxing the system either deliberately or by runaway automated processes. -Rate limiting can be configured on a tier level with tier 0 being reserved for guest users and tiers 1-any for authenticated users. -Superuser accounts are exempt from rate limiting. -Rate limits can be imposed on command APIs by configuring the tier, the command, and the hourly limit in the database. -Two database settings configure the rate limiting. -Note: If either of these settings exist in the database rate limiting will be enabled. -If neither setting exists rate limiting is disabled. - -`:RateLimitingDefaultCapacityTiers` is a comma separated list of default values for each tier. -In the following example, the default for tier `0` (guest users) is set to 10,000 calls per command per hour and tier `1` (authenticated users) is set to 20,000 calls per command per hour. -Tiers not specified in this setting will default to `-1` (No Limit). I.e., -d "10000" is equivalent to -d "10000,-1,-1,..." -`curl http://localhost:8080/api/admin/settings/:RateLimitingDefaultCapacityTiers -X PUT -d '10000,20000'` - -`:RateLimitingCapacityByTierAndAction` is a JSON object specifying the rate by tier and a list of actions (commands). -This allows for more control over the rate limit of individual API command calls. -In the following example, calls made by a guest user (tier 0) for API `GetLatestPublishedDatasetVersionCommand` is further limited to only 10 calls per hour, while an authenticated user (tier 1) will be able to make 30 calls per hour to the same API. -`curl http://localhost:8080/api/admin/settings/:RateLimitingCapacityByTierAndAction -X PUT -d '[{"tier": 0, "limitPerHour": 10, "actions": ["GetLatestPublishedDatasetVersionCommand", "GetPrivateUrlCommand", "GetDatasetCommand", "GetLatestAccessibleDatasetVersionCommand"]}, {"tier": 0, "limitPerHour": 1, "actions": ["CreateGuestbookResponseCommand", "UpdateDatasetVersionCommand", "DestroyDatasetCommand", "DeleteDataFileCommand", "FinalizeDatasetPublicationCommand", "PublishDatasetCommand"]}, {"tier": 1, "limitPerHour": 30, "actions": ["CreateGuestbookResponseCommand", "GetLatestPublishedDatasetVersionCommand", "GetPrivateUrlCommand", "GetDatasetCommand", "GetLatestAccessibleDatasetVersionCommand", "UpdateDatasetVersionCommand", "DestroyDatasetCommand", "DeleteDataFileCommand", "FinalizeDatasetPublicationCommand", "PublishDatasetCommand"]}]'` - -Hazelcast is configured in Payara and should not need any changes for this feature - ### Container Guide, Documentation for Faster Redeploy In the Container Guide, documentation for developers on how to quickly redeploy code has been added for Netbeans and improved for IntelliJ. @@ -139,11 +195,11 @@ Listing collection/dataverse role assignments via API still requires ManageDatav This release adds two missing database constraints that will assure that the externalvocabularyvalue table only has one entry for each uri and that the oaiset table only has one set for each spec. (In the very unlikely case that your existing database has duplicate entries now, install would fail. This can be checked by running -SELECT uri, count(*) FROM externalvocabularyvaluet group by uri; +`SELECT uri, count(*) FROM externalvocabularyvaluet group by uri;` and -SELECT spec, count(*) FROM oaiset group by spec; +`SELECT spec, count(*) FROM oaiset group by spec;` and then removing any duplicate rows (where count>1). @@ -156,9 +212,7 @@ The API endpoint `api/harvest/clients/{harvestingClientNickname}` has been exten - `allowHarvestingMissingCVV`: enable/disable allowing datasets to be harvested with Controlled Vocabulary Values that existed in the originating Dataverse Project but are not in the harvesting Dataverse Project. Default is false. Note: This setting is only available to the API and not currently accessible/settable via the UI -### New QA Guide -A new QA Guide is intended mostly for the core development team but may be of interest to contributors. ### New Accounts Metrics API @@ -230,13 +284,6 @@ In version 6.1, the publication status facet location was unintentionally moved The permissions required to assign a role have been fixed. It is no longer possible to assign a role that includes permissions that the assigning user doesn't have. -### Binder Redirect - -If your installation is configured to use Binder, you should remove the old "girder_ythub" tool and replace it with the tool described at https://github.com/IQSS/dataverse-binder-redirect - -For more information, see #10360. - - -### Optional Croissant Exporter Support +### New QA Guide -When a Dataverse installation is configured to use a metadata exporter for the [Croissant](https://github.com/mlcommons/croissant) format, the content of the JSON-LD in the `` of dataset landing pages will be replaced with that format. However, both JSON-LD and Croissant will still be available for download from the dataset page and API. +A new QA Guide is intended mostly for the core development team but may be of interest to contributors. From d76a59072ca5d4390fdd920aeea2f9f6cee313a6 Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva Date: Tue, 26 Mar 2024 14:56:21 -0400 Subject: [PATCH 04/61] Update --- doc/release-notes/6.2-release-notes.md | 202 ++++++++++++------------- 1 file changed, 101 insertions(+), 101 deletions(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index 57f188bdd20..f3f892ce4e2 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -5,7 +5,7 @@ Please note: To read these instructions in full, please go to https://github.com This release brings new features, enhancements, and bug fixes to the Dataverse software. Thank you to all of the community members who contributed code, suggestions, bug reports, and other assistance across the project. -## Release highlights +## 💡Release highlights ### Return to Author Now Requires a Reason @@ -27,31 +27,31 @@ and will be required in a future version. New microprofile settings (where * indicates a provider id indicating which provider the setting is for): -- `dataverse.pid.providers` -- `dataverse.pid.default-provider` -- `dataverse.pid.*.type` -- `dataverse.pid.*.label` -- `dataverse.pid.*.authority` -- `dataverse.pid.*.shoulder` -- `dataverse.pid.*.identifier-generation-style` -- `dataverse.pid.*.datafile-pid-format` -- `dataverse.pid.*.managed-list` -- `dataverse.pid.*.excluded-list` -- `dataverse.pid.*.datacite.mds-api-url` -- `dataverse.pid.*.datacite.rest-api-url` -- `dataverse.pid.*.datacite.username` -- `dataverse.pid.*.datacite.password` -- `dataverse.pid.*.ezid.api-url` -- `dataverse.pid.*.ezid.username` -- `dataverse.pid.*.ezid.password` -- `dataverse.pid.*.permalink.base-url` -- `dataverse.pid.*.permalink.separator` -- `dataverse.pid.*.handlenet.index` -- `dataverse.pid.*.handlenet.independent-service` -- `dataverse.pid.*.handlenet.auth-handle` -- `dataverse.pid.*.handlenet.key.path` -- `dataverse.pid.*.handlenet.key.passphrase` -- `dataverse.spi.pidproviders.directory` +> - dataverse.pid.providers +> - dataverse.pid.default-provider +> - dataverse.pid.*.type +> - dataverse.pid.*.label +> - dataverse.pid.*.authority +> - dataverse.pid.*.shoulder +> - dataverse.pid.*.identifier-generation-style +> - dataverse.pid.*.datafile-pid-format +> - dataverse.pid.*.managed-list +> - dataverse.pid.*.excluded-list +> - dataverse.pid.*.datacite.mds-api-url +> - dataverse.pid.*.datacite.rest-api-url +> - dataverse.pid.*.datacite.username +> - dataverse.pid.*.datacite.password +> - dataverse.pid.*.ezid.api-url +> - dataverse.pid.*.ezid.username +> - dataverse.pid.*.ezid.password +> - dataverse.pid.*.permalink.base-url +> - dataverse.pid.*.permalink.separator +> - dataverse.pid.*.handlenet.index +> - dataverse.pid.*.handlenet.independent-service +> - dataverse.pid.*.handlenet.auth-handle +> - dataverse.pid.*.handlenet.key.path +> - dataverse.pid.*.handlenet.key.passphrase +> - dataverse.spi.pidproviders.directory ### Rate Limiting Using JCache (With Hazelcast As Provided by Payara) @@ -67,13 +67,6 @@ If neither setting exists rate limiting is disabled. In the following example, the default for tier `0` (guest users) is set to 10,000 calls per command per hour and tier `1` (authenticated users) is set to 20,000 calls per command per hour. Tiers not specified in this setting will default to `-1` (No Limit). I.e., -d "10000" is equivalent to -d "10000,-1,-1,..." -`'curl http://localhost:8080/api/admin/settings/:RateLimitingDefaultCapacityTiers -X PUT -d '10000,20000'` - -`:RateLimitingCapacityByTierAndAction` is a JSON object specifying the rate by tier and a list of actions (commands). -This allows for more control over the rate limit of individual API command calls. -In the following example, calls made by a guest user (tier 0) for API `GetLatestPublishedDatasetVersionCommand` is further limited to only 10 calls per hour, while an authenticated user (tier 1) will be able to make 30 calls per hour to the same API. - -`curl http://localhost:8080/api/admin/settings/:RateLimitingCapacityByTierAndAction -X PUT -d '[{"tier": 0, "limitPerHour": 10, "actions": ["GetLatestPublishedDatasetVersionCommand", "GetPrivateUrlCommand", "GetDatasetCommand", "GetLatestAccessibleDatasetVersionCommand"]}, {"tier": 0, "limitPerHour": 1, "actions": ["CreateGuestbookResponseCommand", "UpdateDatasetVersionCommand", "DestroyDatasetCommand", "DeleteDataFileCommand", "FinalizeDatasetPublicationCommand", "PublishDatasetCommand"]}, {"tier": 1, "limitPerHour": 30, "actions": ["CreateGuestbookResponseCommand", "GetLatestPublishedDatasetVersionCommand", "GetPrivateUrlCommand", "GetDatasetCommand", "GetLatestAccessibleDatasetVersionCommand", "UpdateDatasetVersionCommand", "DestroyDatasetCommand", "DeleteDataFileCommand", "FinalizeDatasetPublicationCommand", "PublishDatasetCommand"]}]'` ``` @@ -117,33 +110,29 @@ curl http://localhost:8080/api/admin/settings/:RateLimitingCapacityByTierAndActi }]' ``` +Hazelcast is configured in Payara and should not need any changes for this feature + ### Binder Redirect If your installation is configured to use Binder, you should remove the old "girder_ythub" tool and replace it with the tool described at https://github.com/IQSS/dataverse-binder-redirect For more information, see [#10360](https://github.com/IQSS/dataverse/issues/10360). -### Optional Croissant Exporter Support - -When a Dataverse installation is configured to use a metadata exporter for the [Croissant](https://github.com/mlcommons/croissant) format, the content of the JSON-LD in the `` of dataset landing pages will be replaced with that format. However, both JSON-LD and Croissant will still be available for download from the dataset page and API. +### Optional Croissant 🥐 Exporter Support - - -Hazelcast is configured in Payara and should not need any changes for this feature +When a Dataverse installation is configured to use a metadata exporter for the [Croissant](https://github.com/mlcommons/croissant) format, the content of the JSON-LD in the **<head>** of dataset landing pages will be replaced with that format. However, both JSON-LD and Croissant will still be available for download from the dataset page and API. ### Search by License A new search facet called "License" has been added and will be displayed as long as there is more than one license in datasets and datafiles in browse/search results. This facet allow you to filter by license such as CC0, etc. -Also, the Search API now handles license filtering using the `fq` parameter, for example : `/api/search?q=*&fq=license%3A%22CC0+1.0%22` for CC0 1.0. See PR #10204 - -### New API Endpoint for Clearing an Individual Dataset From Solr +Also, the Search API now handles license filtering using the `fq` parameter, for example : `/api/search?q=*&fq=license%3A%22CC0+1.0%22` for CC0 1.0. -A new Index API endpoint has been added allowing an admin to clear an individual dataset from Solr. +For more information, see [#10204](https://github.com/IQSS/dataverse/issues/10204). ### Add .QPJ and .QMD Extensions to Shapefile Handling -- Support for `.qpj` and `.qmd` files in shapefile uploads has been introduced, ensuring that these files are properly recognized and handled as part of geospatial datasets in Dataverse. +- Support for **.qpj** and **.qmd** files in shapefile uploads has been introduced, ensuring that these files are properly recognized and handled as part of geospatial datasets in Dataverse. ### Ingested Tabular Data Files Can Be Stored Without the Variable Name Header @@ -151,35 +140,59 @@ Tabular Data Ingest can now save the generated archival files with the list of v Access API will be able to take advantage of Direct Download for tab. files saved with these headers on S3 - since they no longer have to be generated and added to the streamed content on the fly. -This behavior is controlled by the new setting `:StoreIngestedTabularFilesWithVarHeaders`. It is false by default, preserving the legacy behavior. When enabled, Dataverse will be able to handle both the newly ingested files, and any already-existing legacy files stored without these headers transparently to the user. E.g. the access API will continue delivering tab-delimited files **with** this header line, whether it needs to add it dynamically for the legacy files, or reading complete files directly from storage for the ones stored with it. +This behavior is controlled by the new setting **:StoreIngestedTabularFilesWithVarHeaders**. It is false by default, preserving the legacy behavior. When enabled, Dataverse will be able to handle both the newly ingested files, and any already-existing legacy files stored without these headers transparently to the user. E.g. the access API will continue delivering tab-delimited files **with** this header line, whether it needs to add it dynamically for the legacy files, or reading complete files directly from storage for the ones stored with it. An API for converting existing legacy tabular files will be added separately. [this line will need to be changed if we have time to add said API before 6.2 is released]. [TODO] +### Uningest/Reingest Options Available in the File Page Edit Menu + +New Uningest/Reingest options are available in the File Page Edit menu. Ingest errors can be cleared by users who can published the associated dataset and by superusers, allowing for a successful ingest to be undone or retried (e.g. after a Dataverse version update or if ingest size limits are changed). + +The /api/files//uningest api also now allows users who can publish the dataset to undo an ingest failure. + +## 🪲 Bugs + +### Publication Status Facet Restored + +In version 6.1, the publication status facet location was unintentionally moved to the bottom. In this version, we have restored the original order. + +### Permissions Required To Assign a Role Have Been Fixed + +The permissions required to assign a role have been fixed. It is no longer possible to assign a role that includes permissions that the assigning user doesn't have. + ### Geospatial Metadata Block Fields for North and South Renamed The Geospatial metadata block fields for north and south were labeled incorrectly as ‘Longitudes,’ as reported on #5645. After updating to this version of Dataverse, users will need to update all the endpoints that used ‘northLongitude’ and ‘southLongitude’ to ‘northLatitude’ and ‘southLatitude,’ respectively. - TODO: Whoever puts the release notes together should make sure there is the standard note about updating the schema after upgrading. ### OAI-PMH Error Handling Has Been Improved OAI-PMH error handling has been improved to display a machine-readable error in XML rather than a 500 error with no further information. -- /oai?foo=bar will show "No argument 'verb' found" -- /oai?verb=foo&verb=bar will show "Verb must be singular, given: '[foo, bar]'" +> - /oai?foo=bar will show "No argument 'verb' found" +> - /oai?verb=foo&verb=bar will show "Verb must be singular, given: '[foo, bar]'" -### Container Guide, Documentation for Faster Redeploy +## 💾 Persistence -In the Container Guide, documentation for developers on how to quickly redeploy code has been added for Netbeans and improved for IntelliJ. +### Missing Database Constraints -Also in the context of containers, a new option to skip deployment has been added and the war file is now consistently named "dataverse.war" rather than having a version in the filename, such as "dataverse-6.1.war". This predictability makes tooling easier. +This release adds two missing database constraints that will assure that the externalvocabularyvalue table only has one entry for each uri and that the oaiset table only has one set for each spec. (In the very unlikely case that your existing database has duplicate entries now, install would fail. This can be checked by running -Finally, an option to create tabs in the guides using [Sphinx Tabs](https://sphinx-tabs.readthedocs.io) has been added. (You can see the tabs in action in the "dev usage" page of the Container Guide.) To continue building the guides, you will need to install this new dependency by re-running `pip install -r requirements.txt`. +``` +SELECT uri, count(*) FROM externalvocabularyvaluet group by uri; +``` +and +``` +SELECT spec, count(*) FROM oaiset group by spec; +``` +and then removing any duplicate rows (where count>1). + +TODO: Whoever puts the release notes together should make sure there is the standard note about reloading metadata blocks for the citation, astrophysics, and biomedical blocks (plus any others from other PRs) after upgrading. ### Universe Field in Variablemetadata Table Changed -Universe field in variablemetadata table was changed from varchar(255) to text. The change was made to support longer strings in "universe" metadata field, similar to the rest of text fields in variablemetadata table. +Universe field in variablemetadata table was changed from **varchar(255)** to **text**. The change was made to support longer strings in "universe" metadata field, similar to the rest of text fields in variablemetadata table. ### Postgres Versions @@ -187,80 +200,67 @@ This release adds install script support for the new permissions model in Postgr Postgres 13 remains the version used with automated testing. +## 🌐 API + ### Listing Collection/Dataverse API Listing collection/dataverse role assignments via API still requires ManageDataversePermissions, but listing dataset role assignments via API now requires only ManageDatasetPermissions. -### Missing Database Constraints +### New API Endpoint for Clearing an Individual Dataset From Solr -This release adds two missing database constraints that will assure that the externalvocabularyvalue table only has one entry for each uri and that the oaiset table only has one set for each spec. (In the very unlikely case that your existing database has duplicate entries now, install would fail. This can be checked by running +A new Index API endpoint has been added allowing an admin to clear an individual dataset from Solr. -`SELECT uri, count(*) FROM externalvocabularyvaluet group by uri;` +### New Accounts Metrics API -and +Users can retrieve new types of metrics related to user accounts. The new capabilities are [described](https://guides.dataverse.org/en/6.2/api/metrics.html) in the guides. -`SELECT spec, count(*) FROM oaiset group by spec;` +### New canDownloadAtLeastOneFile Endpoint -and then removing any duplicate rows (where count>1). +The `/api/datasets/{id}/versions/{versionId}/canDownloadAtLeastOneFile` endpoint has been created. -TODO: Whoever puts the release notes together should make sure there is the standard note about reloading metadata blocks for the citation, astrophysics, and biomedical blocks (plus any others from other PRs) after upgrading. +This API endpoint indicates if the calling user can download at least one file from a dataset version. Note that Shibboleth group permissions are not considered. -### Harvesting Client API +### Harvesting Client Endpoint Extended The API endpoint `api/harvest/clients/{harvestingClientNickname}` has been extended to include the following fields: -- `allowHarvestingMissingCVV`: enable/disable allowing datasets to be harvested with Controlled Vocabulary Values that existed in the originating Dataverse Project but are not in the harvesting Dataverse Project. Default is false. -Note: This setting is only available to the API and not currently accessible/settable via the UI - - - -### New Accounts Metrics API - -Users can retrieve new types of metrics related to user accounts. The new capabilities are [described](https://guides.dataverse.org/en/6.2/api/metrics.html) in the guides. - -### New canDownloadAtLeastOneFile API +- **allowHarvestingMissingCVV**: enable/disable allowing datasets to be harvested with Controlled Vocabulary Values that existed in the originating Dataverse Project but are not in the harvesting Dataverse Project. Default is false. -The GET canDownloadAtLeastOneFile (/api/datasets/{id}/versions/{versionId}/canDownloadAtLeastOneFile) endpoint has been created. - -This API endpoint indicates if the calling user can download at least one file from a dataset version. Note that Shibboleth group permissions are not considered. +*Note: This setting is only available to the API and not currently accessible/settable via the UI* -### Extended getVersionFiles API +### Version Files Endpoint Extended -The response for getVersionFiles (/api/datasets/{id}/versions/{versionId}/files) endpoint has been modified to include a total count of records available (totalCount:x). +The response for getVersionFiles `/api/datasets/{id}/versions/{versionId}/files` endpoint has been modified to include a total count of records available **totalCount:x**. This will aid in pagination by allowing the caller to know how many pages can be iterated through. The existing API (getVersionFileCounts) to return the count will still be available. -### Extended Metadata Blocks API +### Metadata Blocks Endpoint Extended The API endpoint `/api/metadatablocks/{block_id}` has been extended to include the following fields: -- `isRequired`: Whether or not this field is required -- `displayOrder`: The display order of the field in create/edit forms -- `typeClass`: The type class of this field ("controlledVocabulary", "compound", or "primitive") - -### Evaluation Version Tutorial on the Containers Guide - -The Container Guide now containers a tutorial for running Dataverse in containers for demo or evaluation purposes: https://guides.dataverse.org/en/6.2/container +- **isRequired**: Whether or not this field is required +- **displayOrder**: The display order of the field in create/edit forms +- **typeClass**: The type class of this field ("controlledVocabulary", "compound", or "primitive") ### Get File Citation As JSON It is now possible to retrieve via API the file citation as it appears on the file landing page. It is formatted in HTML and encoded in JSON. -This API is not for downloading various citation formats such as EndNote XML, RIS, or BibTeX. This functionality has been requested in https://github.com/IQSS/dataverse/issues/3140 and https://github.com/IQSS/dataverse/issues/9994 +This API is not for downloading various citation formats such as EndNote XML, RIS, or BibTeX. This functionality has been requested in [#3140](https://github.com/IQSS/dataverse/issues/3140) and [#9994](https://github.com/IQSS/dataverse/issues/9994) -### Extended Files API +### Files Endpoint Extended The API endpoint `api/files/{id}` has been extended to support the following optional query parameters: -- `includeDeaccessioned`: Indicates whether or not to consider deaccessioned dataset versions in the latest file search. (Default: `false`). -- `returnDatasetVersion`: Indicates whether or not to include the dataset version of the file in the response. (Default: `false`). +- **includeDeaccessioned**: Indicates whether or not to consider deaccessioned dataset versions in the latest file search. (Default: `false`). +- **returnDatasetVersion**: Indicates whether or not to include the dataset version of the file in the response. (Default: `false`). -A new endpoint `api/files/{id}/versions/{datasetVersionId}` has been created. This endpoint returns the file metadata present in the requested dataset version. To specify the dataset version, you can use ``:latest-published``, or ``:latest``, or ``:draft`` or ``1.0`` or any other available version identifier. +A new endpoint `api/files/{id}/versions/{datasetVersionId}` has been created. This endpoint returns the file metadata present in the requested dataset version. To specify the dataset version, you can use `:latest-published`, `:latest`, `:draft` or `1.0` or any other available version identifier. -The endpoint supports the `includeDeaccessioned` and `returnDatasetVersion` optional query parameters, as does the `api/files/{id}` endpoint. +The endpoint supports the *includeDeaccessioned* and *returnDatasetVersion* optional query parameters, as does the `api/files/{id}` endpoint. `api/files/{id}/draft` endpoint is no longer available in favor of the new endpoint `api/files/{id}/versions/{datasetVersionId}`, which can use the version identifier ``:draft`` (`api/files/{id}/versions/:draft`) to obtain the same result. -### Endpoint Extended: Datasets, Dataverse Collections, and Datafiles +### Datasets, Dataverse Collections, and Datafiles Endpoints Extended The API endpoints for getting datasets, Dataverse collections, and datafiles have been extended to support the following optional 'returnOwners' query parameter. @@ -270,20 +270,20 @@ Including the parameter and setting it to true will add a hierarchy showing whic The API endpoint `api/datasets/{id}/metadata` has been changed to default to the latest version of the dataset that the user has access. -### Uningest/Reingest Options Available in the File Page Edit Menu +## 📖 Guides -New Uningest/Reingest options are available in the File Page Edit menu, allowing ingest errors to be cleared (by users who can published the associated dataset) -and (by superusers) for a successful ingest to be undone or retried (e.g. after a Dataverse version update or if ingest size limits are changed). -The /api/files//uningest api also now allows users who can publish the dataset to undo an ingest failure. +### New QA Guide -### Publication Status Facet Restored +A new QA Guide is intended mostly for the core development team but may be of interest to contributors. -In version 6.1, the publication status facet location was unintentionally moved to the bottom. In this version, we have restored the original order. +### Container Guide, Documentation for Faster Redeploy -### Permissions Required To Assign a Role Have Been Fixed +In the Container Guide, documentation for developers on how to quickly redeploy code has been added for Netbeans and improved for IntelliJ. -The permissions required to assign a role have been fixed. It is no longer possible to assign a role that includes permissions that the assigning user doesn't have. +Also in the context of containers, a new option to skip deployment has been added and the war file is now consistently named "dataverse.war" rather than having a version in the filename, such as "dataverse-6.1.war". This predictability makes tooling easier. -### New QA Guide +Finally, an option to create tabs in the guides using [Sphinx Tabs](https://sphinx-tabs.readthedocs.io) has been added. (You can see the tabs in action in the "dev usage" page of the Container Guide.) To continue building the guides, you will need to install this new dependency by re-running `pip install -r requirements.txt`. -A new QA Guide is intended mostly for the core development team but may be of interest to contributors. +### Evaluation Version Tutorial on the Containers Guide + +The Container Guide now containers a tutorial for running Dataverse in containers for demo or evaluation purposes: https://guides.dataverse.org/en/6.2/container \ No newline at end of file From ccc839d7ba492cf511c217765c0c5f60e1ea2f1e Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva Date: Tue, 26 Mar 2024 14:58:48 -0400 Subject: [PATCH 05/61] Spacing --- doc/release-notes/6.2-release-notes.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index f3f892ce4e2..7eb162c20bc 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -5,6 +5,7 @@ Please note: To read these instructions in full, please go to https://github.com This release brings new features, enhancements, and bug fixes to the Dataverse software. Thank you to all of the community members who contributed code, suggestions, bug reports, and other assistance across the project. + ## 💡Release highlights ### Return to Author Now Requires a Reason @@ -67,8 +68,6 @@ If neither setting exists rate limiting is disabled. In the following example, the default for tier `0` (guest users) is set to 10,000 calls per command per hour and tier `1` (authenticated users) is set to 20,000 calls per command per hour. Tiers not specified in this setting will default to `-1` (No Limit). I.e., -d "10000" is equivalent to -d "10000,-1,-1,..." - - ``` curl http://localhost:8080/api/admin/settings/:RateLimitingCapacityByTierAndAction -X PUT -d '[{ "tier": 0, @@ -150,6 +149,7 @@ New Uningest/Reingest options are available in the File Page Edit menu. Ingest e The /api/files//uningest api also now allows users who can publish the dataset to undo an ingest failure. + ## 🪲 Bugs ### Publication Status Facet Restored @@ -173,6 +173,7 @@ OAI-PMH error handling has been improved to display a machine-readable error in > - /oai?foo=bar will show "No argument 'verb' found" > - /oai?verb=foo&verb=bar will show "Verb must be singular, given: '[foo, bar]'" + ## 💾 Persistence ### Missing Database Constraints @@ -200,6 +201,7 @@ This release adds install script support for the new permissions model in Postgr Postgres 13 remains the version used with automated testing. + ## 🌐 API ### Listing Collection/Dataverse API @@ -270,6 +272,7 @@ Including the parameter and setting it to true will add a hierarchy showing whic The API endpoint `api/datasets/{id}/metadata` has been changed to default to the latest version of the dataset that the user has access. + ## 📖 Guides ### New QA Guide From bcddcf6a9f4ae4b4bd9048aa702019b5110264d5 Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva Date: Tue, 26 Mar 2024 14:59:48 -0400 Subject: [PATCH 06/61] Bugs --- doc/release-notes/6.2-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index 7eb162c20bc..70b64c15550 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -150,7 +150,7 @@ New Uningest/Reingest options are available in the File Page Edit menu. Ingest e The /api/files//uningest api also now allows users who can publish the dataset to undo an ingest failure. -## 🪲 Bugs +## 🪲 Bug fixes ### Publication Status Facet Restored From 09d746d9cf2d9fa9558016a8f9194c8fdfa7d4bd Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva Date: Tue, 26 Mar 2024 16:20:43 -0400 Subject: [PATCH 07/61] 7424 Updated --- doc/release-notes/6.2-release-notes.md | 25 +++++++++++++++++++++++++ doc/release-notes/7424-mailsession.md | 24 ------------------------ 2 files changed, 25 insertions(+), 24 deletions(-) delete mode 100644 doc/release-notes/7424-mailsession.md diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index 70b64c15550..519f342b2d0 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -111,6 +111,31 @@ curl http://localhost:8080/api/admin/settings/:RateLimitingCapacityByTierAndActi Hazelcast is configured in Payara and should not need any changes for this feature +## Simplified SMTP configuration + +With this release, we deprecate the usage of `asadmin create-javamail-resource` to configure Dataverse to send mail using your SMTP server and provide a simplified, standard alternative using JVM options or MicroProfile Config. + +At this point, no action is required if you want to keep your current configuration. +Warnings will show in your server logs to inform and remind you about the deprecation. +A future major release of Dataverse may remove this way of configuration. + +Please do take the opportunity to update your SMTP configuration. Details can be found in section of the Installation Guide starting with the [SMTP/Email Configuration](https://guides.dataverse.org/en/6.2/installation/config.html#smtp-email-configuration) section of the Installation Guide. + +Once reconfiguration is complete, you should remove legacy, unused config. First, run `asadmin delete-javamail-resource mail/notifyMailSession` as described in the [6.1 guides](https://guides.dataverse.org/en/6.1/installation/installation-main.html#mail-host-configuration-authentication). Then run `curl -X DELETE http://localhost:8080/api/admin/settings/:SystemEmail` as this database setting has been replace with `dataverse.mail.system-email` as described below. + +Please note: as there have been problems with email delivered to SPAM folders when the "From" within mail envelope and the mail session configuration didn't match (#4210), as of this version the sole source for the "From" address is the setting `dataverse.mail.system-email` once you migrate to the new way of configuration. + +List of options added: +> - dataverse.mail.system-email +> - dataverse.mail.mta.host +> - dataverse.mail.mta.port +> - dataverse.mail.mta.ssl.enable +> - dataverse.mail.mta.auth +> - dataverse.mail.mta.user +> - dataverse.mail.mta.password +> - dataverse.mail.mta.allow-utf8-addresses +> - Plus many more for advanced usage and special provider requirements. See [configuration guide for a full list](https://guides.dataverse.org/en/6.2/installation/config.html#dataverse-mail-mta). + ### Binder Redirect If your installation is configured to use Binder, you should remove the old "girder_ythub" tool and replace it with the tool described at https://github.com/IQSS/dataverse-binder-redirect diff --git a/doc/release-notes/7424-mailsession.md b/doc/release-notes/7424-mailsession.md deleted file mode 100644 index 67c876f7ad5..00000000000 --- a/doc/release-notes/7424-mailsession.md +++ /dev/null @@ -1,24 +0,0 @@ -## Simplified SMTP configuration - -With this release, we deprecate the usage of `asadmin create-javamail-resource` to configure Dataverse to send mail using your SMTP server and provide a simplified, standard alternative using JVM options or MicroProfile Config. - -At this point, no action is required if you want to keep your current configuration. -Warnings will show in your server logs to inform and remind you about the deprecation. -A future major release of Dataverse may remove this way of configuration. - -Please do take the opportunity to update your SMTP configuration. Details can be found in section of the Installation Guide starting with the [SMTP/Email Configuration](https://guides.dataverse.org/en/6.2/installation/config.html#smtp-email-configuration) section of the Installation Guide. - -Once reconfiguration is complete, you should remove legacy, unused config. First, run `asadmin delete-javamail-resource mail/notifyMailSession` as described in the [6.1 guides](https://guides.dataverse.org/en/6.1/installation/installation-main.html#mail-host-configuration-authentication). Then run `curl -X DELETE http://localhost:8080/api/admin/settings/:SystemEmail` as this database setting has been replace with `dataverse.mail.system-email` as described below. - -Please note: as there have been problems with email delivered to SPAM folders when the "From" within mail envelope and the mail session configuration didn't match (#4210), as of this version the sole source for the "From" address is the setting `dataverse.mail.system-email` once you migrate to the new way of configuration. - -List of options added: -- dataverse.mail.system-email -- dataverse.mail.mta.host -- dataverse.mail.mta.port -- dataverse.mail.mta.ssl.enable -- dataverse.mail.mta.auth -- dataverse.mail.mta.user -- dataverse.mail.mta.password -- dataverse.mail.mta.allow-utf8-addresses -- Plus many more for advanced usage and special provider requirements. See [configuration guide for a full list](https://guides.dataverse.org/en/6.2/installation/config.html#dataverse-mail-mta). \ No newline at end of file From 06c82a8cefc390bd0742e453bf2b82853bbeccbe Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva <142103991+jp-tosca@users.noreply.github.com> Date: Wed, 27 Mar 2024 09:09:10 -0400 Subject: [PATCH 08/61] Update doc/release-notes/6.2-release-notes.md Co-authored-by: Philip Durbin --- doc/release-notes/6.2-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index 519f342b2d0..3fdb25d8bfb 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -111,7 +111,7 @@ curl http://localhost:8080/api/admin/settings/:RateLimitingCapacityByTierAndActi Hazelcast is configured in Payara and should not need any changes for this feature -## Simplified SMTP configuration +### Simplified SMTP configuration With this release, we deprecate the usage of `asadmin create-javamail-resource` to configure Dataverse to send mail using your SMTP server and provide a simplified, standard alternative using JVM options or MicroProfile Config. From 3b7e729d5092ed43dfb12f4a6829ed87d2871852 Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva Date: Wed, 27 Mar 2024 09:45:18 -0400 Subject: [PATCH 09/61] First round changes --- doc/release-notes/6.2-release-notes.md | 85 +++++++++++++++----------- 1 file changed, 49 insertions(+), 36 deletions(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index 3fdb25d8bfb..5e211f96284 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -5,8 +5,15 @@ Please note: To read these instructions in full, please go to https://github.com This release brings new features, enhancements, and bug fixes to the Dataverse software. Thank you to all of the community members who contributed code, suggestions, bug reports, and other assistance across the project. +### Search and Facet by License -## 💡Release highlights +A new search facet called "License" has been added and will be displayed as long as there is more than one license in datasets and datafiles in browse/search results. This facet allow you to filter by license such as CC0, etc. + +Also, the Search API now handles license filtering using the `fq` parameter, for example : `/api/search?q=*&fq=license%3A%22CC0+1.0%22` for CC0 1.0. + +For more information, see [#10204](https://github.com/IQSS/dataverse/issues/10204). + +## 💡Release Highlights ### Return to Author Now Requires a Reason @@ -14,9 +21,10 @@ The Popup for returning to author now requires a reason that will be sent by ema Please note that you can still type a creative and meaningful comment such as "The author would like to modify his dataset", "Files are missing", "Nothing to report" or "A curation report with comments and suggestions/instructions will follow in another email" that suits your situation. + ### Support for Using Multiple PID Providers -This release adds support for using multiple PID (DOI, Handle, PermalLink) providers, multiple PID provider accounts +This release adds support for using multiple PID (DOI, Handle, PermaLink) providers, multiple PID provider accounts (managing a given protocol, authority,separator, shoulder combination), assigning PID provider accounts to specific collections, and supporting transferred PIDs (where a PID is managed by an account when it's authority, separator, and/or shoulder don't match the combination where the account can mint new PIDs). It also adds the ability for additional provider services beyond the existing @@ -26,33 +34,8 @@ These changes require per-provider settings rather than the global PID settings for installations using a single PID Provider account is provided, updating to use the new microprofile settings is highly recommended and will be required in a future version. -New microprofile settings (where * indicates a provider id indicating which provider the setting is for): +[New microprofile settings](#new-microprofile-settings) -> - dataverse.pid.providers -> - dataverse.pid.default-provider -> - dataverse.pid.*.type -> - dataverse.pid.*.label -> - dataverse.pid.*.authority -> - dataverse.pid.*.shoulder -> - dataverse.pid.*.identifier-generation-style -> - dataverse.pid.*.datafile-pid-format -> - dataverse.pid.*.managed-list -> - dataverse.pid.*.excluded-list -> - dataverse.pid.*.datacite.mds-api-url -> - dataverse.pid.*.datacite.rest-api-url -> - dataverse.pid.*.datacite.username -> - dataverse.pid.*.datacite.password -> - dataverse.pid.*.ezid.api-url -> - dataverse.pid.*.ezid.username -> - dataverse.pid.*.ezid.password -> - dataverse.pid.*.permalink.base-url -> - dataverse.pid.*.permalink.separator -> - dataverse.pid.*.handlenet.index -> - dataverse.pid.*.handlenet.independent-service -> - dataverse.pid.*.handlenet.auth-handle -> - dataverse.pid.*.handlenet.key.path -> - dataverse.pid.*.handlenet.key.passphrase -> - dataverse.spi.pidproviders.directory ### Rate Limiting Using JCache (With Hazelcast As Provided by Payara) @@ -125,7 +108,8 @@ Once reconfiguration is complete, you should remove legacy, unused config. First Please note: as there have been problems with email delivered to SPAM folders when the "From" within mail envelope and the mail session configuration didn't match (#4210), as of this version the sole source for the "From" address is the setting `dataverse.mail.system-email` once you migrate to the new way of configuration. -List of options added: +For a list of new settings see the section below: + > - dataverse.mail.system-email > - dataverse.mail.mta.host > - dataverse.mail.mta.port @@ -146,13 +130,7 @@ For more information, see [#10360](https://github.com/IQSS/dataverse/issues/1036 When a Dataverse installation is configured to use a metadata exporter for the [Croissant](https://github.com/mlcommons/croissant) format, the content of the JSON-LD in the **<head>** of dataset landing pages will be replaced with that format. However, both JSON-LD and Croissant will still be available for download from the dataset page and API. -### Search by License -A new search facet called "License" has been added and will be displayed as long as there is more than one license in datasets and datafiles in browse/search results. This facet allow you to filter by license such as CC0, etc. - -Also, the Search API now handles license filtering using the `fq` parameter, for example : `/api/search?q=*&fq=license%3A%22CC0+1.0%22` for CC0 1.0. - -For more information, see [#10204](https://github.com/IQSS/dataverse/issues/10204). ### Add .QPJ and .QMD Extensions to Shapefile Handling @@ -314,4 +292,39 @@ Finally, an option to create tabs in the guides using [Sphinx Tabs](https://sphi ### Evaluation Version Tutorial on the Containers Guide -The Container Guide now containers a tutorial for running Dataverse in containers for demo or evaluation purposes: https://guides.dataverse.org/en/6.2/container \ No newline at end of file +The Container Guide now containers a tutorial for running Dataverse in containers for demo or evaluation purposes: https://guides.dataverse.org/en/6.2/container + +*** + +# New Settings + + +### New microprofile settings : [Go back](#multiple-pid-sup) + +*The * indicates a provider id indicating which provider the setting is for* + +> - dataverse.pid.providers +> - dataverse.pid.default-provider +> - dataverse.pid.*.type +> - dataverse.pid.*.label +> - dataverse.pid.*.authority +> - dataverse.pid.*.shoulder +> - dataverse.pid.*.identifier-generation-style +> - dataverse.pid.*.datafile-pid-format +> - dataverse.pid.*.managed-list +> - dataverse.pid.*.excluded-list +> - dataverse.pid.*.datacite.mds-api-url +> - dataverse.pid.*.datacite.rest-api-url +> - dataverse.pid.*.datacite.username +> - dataverse.pid.*.datacite.password +> - dataverse.pid.*.ezid.api-url +> - dataverse.pid.*.ezid.username +> - dataverse.pid.*.ezid.password +> - dataverse.pid.*.permalink.base-url +> - dataverse.pid.*.permalink.separator +> - dataverse.pid.*.handlenet.index +> - dataverse.pid.*.handlenet.independent-service +> - dataverse.pid.*.handlenet.auth-handle +> - dataverse.pid.*.handlenet.key.path +> - dataverse.pid.*.handlenet.key.passphrase +> - dataverse.spi.pidproviders.directory \ No newline at end of file From ca5506a780389f70e9f18d37d4a081659929d156 Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva Date: Wed, 27 Mar 2024 12:20:34 -0400 Subject: [PATCH 10/61] Corrections and instructions for update --- doc/release-notes/6.1-release-notes.md | 3 +- doc/release-notes/6.2-release-notes.md | 162 +++++++++++++++++++++---- 2 files changed, 138 insertions(+), 27 deletions(-) diff --git a/doc/release-notes/6.1-release-notes.md b/doc/release-notes/6.1-release-notes.md index 1279d09a023..dbeda726aad 100644 --- a/doc/release-notes/6.1-release-notes.md +++ b/doc/release-notes/6.1-release-notes.md @@ -247,7 +247,7 @@ Upgrading requires a maintenance window and downtime. Please plan ahead, create These instructions assume that you've already upgraded through all the 5.x releases and are now running Dataverse 6.0. -0\. These instructions assume that you are upgrading from 6.0. If you are running an earlier version, the only safe way to upgrade is to progress through the upgrades to all the releases in between before attempting the upgrade to 5.14. +0\. These instructions assume that you are upgrading from 6.0. If you are running an earlier version, the only safe way to upgrade is to progress through the upgrades to all the releases in between before attempting the upgrade to 6.1. If you are running Payara as a non-root user (and you should be!), **remember not to execute the commands below as root**. Use `sudo` to change to that user first. For example, `sudo -i -u dataverse` if `dataverse` is your dedicated application user. @@ -288,6 +288,7 @@ As noted above, deployment of the war file might take several minutes due a data 6a\. Update Citation Metadata Block (to make Alternative Title repeatable) +- `wget https://github.com/IQSS/dataverse/releases/download/v6.1/citation.tsv` - `curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/tab-separated-values" -X POST --upload-file scripts/api/data/metadatablocks/citation.tsv` 7\. Upate Solr schema.xml to allow multiple Alternative Titles to be used. See specific instructions below for those installations without custom metadata blocks (7a) and those with custom metadata blocks (7b). diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index 5e211f96284..529b8e59075 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -17,16 +17,16 @@ For more information, see [#10204](https://github.com/IQSS/dataverse/issues/1020 ### Return to Author Now Requires a Reason -The Popup for returning to author now requires a reason that will be sent by email to the author. +The Popup for returning to author now allows to type in a message to explain the reasons of return and potential edits needed, that will be sent by email to the author. -Please note that you can still type a creative and meaningful comment such as "The author would like to modify his dataset", "Files are missing", "Nothing to report" or "A curation report with comments and suggestions/instructions will follow in another email" that suits your situation. +Please note that this note is mandatory, but that you can still type a creative and meaningful comment such as "The author would like to modify his dataset", "Files are missing", "Nothing to report" or "A curation report with comments and suggestions/instructions will follow in another email" that suits your situation. ### Support for Using Multiple PID Providers This release adds support for using multiple PID (DOI, Handle, PermaLink) providers, multiple PID provider accounts (managing a given protocol, authority,separator, shoulder combination), assigning PID provider accounts to specific collections, -and supporting transferred PIDs (where a PID is managed by an account when it's authority, separator, and/or shoulder don't match +and supporting transferred PIDs (where a PID is managed by an account when its authority, separator, and/or shoulder don't match the combination where the account can mint new PIDs). It also adds the ability for additional provider services beyond the existing DataCite, EZId, Handle, and PermaLink providers to be dynamically added as separate jar files. @@ -108,17 +108,9 @@ Once reconfiguration is complete, you should remove legacy, unused config. First Please note: as there have been problems with email delivered to SPAM folders when the "From" within mail envelope and the mail session configuration didn't match (#4210), as of this version the sole source for the "From" address is the setting `dataverse.mail.system-email` once you migrate to the new way of configuration. -For a list of new settings see the section below: +[New SMTP settings](#smtp-settings): + -> - dataverse.mail.system-email -> - dataverse.mail.mta.host -> - dataverse.mail.mta.port -> - dataverse.mail.mta.ssl.enable -> - dataverse.mail.mta.auth -> - dataverse.mail.mta.user -> - dataverse.mail.mta.password -> - dataverse.mail.mta.allow-utf8-addresses -> - Plus many more for advanced usage and special provider requirements. See [configuration guide for a full list](https://guides.dataverse.org/en/6.2/installation/config.html#dataverse-mail-mta). ### Binder Redirect @@ -144,7 +136,7 @@ Access API will be able to take advantage of Direct Download for tab. files save This behavior is controlled by the new setting **:StoreIngestedTabularFilesWithVarHeaders**. It is false by default, preserving the legacy behavior. When enabled, Dataverse will be able to handle both the newly ingested files, and any already-existing legacy files stored without these headers transparently to the user. E.g. the access API will continue delivering tab-delimited files **with** this header line, whether it needs to add it dynamically for the legacy files, or reading complete files directly from storage for the ones stored with it. -An API for converting existing legacy tabular files will be added separately. [this line will need to be changed if we have time to add said API before 6.2 is released]. [TODO] +We are planning to add an API for converting existing legacy tabular files in a future release. ### Uningest/Reingest Options Available in the File Page Edit Menu @@ -167,8 +159,6 @@ The permissions required to assign a role have been fixed. It is no longer possi The Geospatial metadata block fields for north and south were labeled incorrectly as ‘Longitudes,’ as reported on #5645. After updating to this version of Dataverse, users will need to update all the endpoints that used ‘northLongitude’ and ‘southLongitude’ to ‘northLatitude’ and ‘southLatitude,’ respectively. -TODO: Whoever puts the release notes together should make sure there is the standard note about updating the schema after upgrading. - ### OAI-PMH Error Handling Has Been Improved OAI-PMH error handling has been improved to display a machine-readable error in XML rather than a 500 error with no further information. @@ -176,6 +166,9 @@ OAI-PMH error handling has been improved to display a machine-readable error in > - /oai?foo=bar will show "No argument 'verb' found" > - /oai?verb=foo&verb=bar will show "Verb must be singular, given: '[foo, bar]'" +### Granting File Access Without Access Request + +A bug introduced with the guestboook-at-request, requests are not deleted when granted, they are now given the state granted. ## 💾 Persistence @@ -186,13 +179,12 @@ This release adds two missing database constraints that will assure that the ext ``` SELECT uri, count(*) FROM externalvocabularyvaluet group by uri; ``` -and +And: ``` SELECT spec, count(*) FROM oaiset group by spec; ``` -and then removing any duplicate rows (where count>1). +Then removing any duplicate rows (where count>1). -TODO: Whoever puts the release notes together should make sure there is the standard note about reloading metadata blocks for the citation, astrophysics, and biomedical blocks (plus any others from other PRs) after upgrading. ### Universe Field in Variablemetadata Table Changed @@ -278,10 +270,6 @@ The API endpoint `api/datasets/{id}/metadata` has been changed to default to the ## 📖 Guides -### New QA Guide - -A new QA Guide is intended mostly for the core development team but may be of interest to contributors. - ### Container Guide, Documentation for Faster Redeploy In the Container Guide, documentation for developers on how to quickly redeploy code has been added for Netbeans and improved for IntelliJ. @@ -294,12 +282,16 @@ Finally, an option to create tabs in the guides using [Sphinx Tabs](https://sphi The Container Guide now containers a tutorial for running Dataverse in containers for demo or evaluation purposes: https://guides.dataverse.org/en/6.2/container +### New QA Guide + +A new QA Guide is intended mostly for the core development team but may be of interest to contributors. + *** -# New Settings +# ⚙️ New Settings -### New microprofile settings : [Go back](#multiple-pid-sup) +### Microprofile settings : *The * indicates a provider id indicating which provider the setting is for* @@ -327,4 +319,122 @@ The Container Guide now containers a tutorial for running Dataverse in container > - dataverse.pid.*.handlenet.auth-handle > - dataverse.pid.*.handlenet.key.path > - dataverse.pid.*.handlenet.key.passphrase -> - dataverse.spi.pidproviders.directory \ No newline at end of file +> - dataverse.spi.pidproviders.directory + +[⬅️ Go back](#multiple-pid-sup) + +## SMTP Settings: + +> - dataverse.mail.system-email +> - dataverse.mail.mta.host +> - dataverse.mail.mta.port +> - dataverse.mail.mta.ssl.enable +> - dataverse.mail.mta.auth +> - dataverse.mail.mta.user +> - dataverse.mail.mta.password +> - dataverse.mail.mta.allow-utf8-addresses +> - Plus many more for advanced usage and special provider requirements. See [configuration guide for a full list](https://guides.dataverse.org/en/6.2/installation/config.html#dataverse-mail-mta). + +[⬅️ Go back](#simplified-smtp-configuration) + +## Upgrade instructions +Upgrading requires a maintenance window and downtime. Please plan ahead, create backups of your database, etc. + +These instructions assume that you've already upgraded through all the 5.x releases and are now running Dataverse 6.1. + +0\. These instructions assume that you are upgrading from the immediate previous version. If you are running an earlier version, the only safe way to upgrade is to progress through the upgrades to all the releases in between before attempting the upgrade to this version. + +If you are running Payara as a non-root user (and you should be!), **remember not to execute the commands below as root**. Use `sudo` to change to that user first. For example, `sudo -i -u dataverse` if `dataverse` is your dedicated application user. + +In the following commands we assume that Payara 6 is installed in `/usr/local/payara6`. If not, adjust as needed. + +`export PAYARA=/usr/local/payara6` + +(or `setenv PAYARA /usr/local/payara6` if you are using a `csh`-like shell) + +1\. Undeploy the previous version. + +- `$PAYARA/bin/asadmin undeploy dataverse-6.0` + +2\. Stop Payara and remove the generated directory + +- `service payara stop` +- `rm -rf $PAYARA/glassfish/domains/domain1/generated` + +3\. Start Payara + +- `service payara start` + +4\. Deploy this version. + +- `$PAYARA/bin/asadmin deploy dataverse-6.1.war` + +As noted above, deployment of the war file might take several minutes due a database migration script required for the new storage quotas feature. + +5\. Restart Payara + +- `service payara stop` +- `service payara start` + +6\. Update Geospatial Metadata Block. + + ``` + wget https://github.com/IQSS/dataverse/releases/download/v6.1/geospatial.tsv + + curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/tab-separated-values" -X POST --upload-file @geospatial.tsv + ``` + +6a\. Update Citation Metadata Block. + +``` +wget https://github.com/IQSS/dataverse/releases/download/v6.2/citation.tsv + +curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/tab-separated-values" -X POST --upload-file scripts/api/data/metadatablocks/citation.tsv +``` + +6b\. Update Astrophysics Metadata Block. + +``` +wget https://github.com/IQSS/dataverse/releases/download/v6.2/astrophysics.tsv + +curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/tab-separated-values" -X POST --upload-file scripts/api/data/metadatablocks/astrophysics.tsv +``` + +6c\. Update Biomedical Metadata Block (to make Alternative Title repeatable) + +``` +wget https://github.com/IQSS/dataverse/releases/download/v6.2/biomedical.tsv + +curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/tab-separated-values" -X POST --upload-file scripts/api/data/metadatablocks/biomedical.tsv +``` + +7\. Upate Solr schema.xml. See specific instructions below for those installations without custom metadata blocks (7a) and those with custom metadata blocks (7b). + +7a\. For installations without custom or experimental metadata blocks: + +- Stop Solr instance (usually `service solr stop`, depending on Solr installation/OS, see the [Installation Guide](https://guides.dataverse.org/en/5.14/installation/prerequisites.html#solr-init-script)) + +- Replace schema.xml + + - `cp /tmp/dvinstall/schema.xml /usr/local/solr/solr-9.3.0/server/solr/collection1/conf` + +- Start Solr instance (usually `service solr start`, depending on Solr/OS) + +7b\. For installations with custom or experimental metadata blocks: + +- Stop Solr instance (usually `service solr stop`, depending on Solr installation/OS, see the [Installation Guide](https://guides.dataverse.org/en/5.14/installation/prerequisites.html#solr-init-script)) + +- There are 2 ways to regenerate the schema: Either by collecting the output of the Dataverse schema API and feeding it to the `update-fields.sh` script that we supply, as in the example below (modify the command lines as needed): +``` + wget https://raw.githubusercontent.com/IQSS/dataverse/master/conf/solr/9.3.0/update-fields.sh + chmod +x update-fields.sh + curl "http://localhost:8080/api/admin/index/solr/schema" | ./update-fields.sh /usr/local/solr/solr-9.3.0/server/solr/collection1/conf/schema.xml +``` +OR, alternatively, you can edit the following line in your schema.xml by hand as follows (to indicate that alternative title is now `multiValued="true"`): +``` + +``` + +- Restart Solr instance (usually `service solr restart` depending on solr/OS) + +8\. Run ReExportAll to update dataset metadata exports. Follow the directions in the [Admin Guide](http://guides.dataverse.org/en/5.14/admin/metadataexport.html#batch-exports-through-the-api). \ No newline at end of file From bda01859486c9a6887f2bf054375aefc4f54e2f9 Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva Date: Wed, 27 Mar 2024 13:16:21 -0400 Subject: [PATCH 11/61] Update with initial feedback --- doc/release-notes/6.2-release-notes.md | 126 +++++++++++++------------ 1 file changed, 66 insertions(+), 60 deletions(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index 529b8e59075..8e5bf32b0e1 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -34,7 +34,7 @@ These changes require per-provider settings rather than the global PID settings for installations using a single PID Provider account is provided, updating to use the new microprofile settings is highly recommended and will be required in a future version. -[New microprofile settings](#new-microprofile-settings) +[New microprofile settings](#microprofile-settings) ### Rate Limiting Using JCache (With Hazelcast As Provided by Payara) @@ -110,8 +110,6 @@ Please note: as there have been problems with email delivered to SPAM folders wh [New SMTP settings](#smtp-settings): - - ### Binder Redirect If your installation is configured to use Binder, you should remove the old "girder_ythub" tool and replace it with the tool described at https://github.com/IQSS/dataverse-binder-redirect @@ -122,11 +120,13 @@ For more information, see [#10360](https://github.com/IQSS/dataverse/issues/1036 When a Dataverse installation is configured to use a metadata exporter for the [Croissant](https://github.com/mlcommons/croissant) format, the content of the JSON-LD in the **<head>** of dataset landing pages will be replaced with that format. However, both JSON-LD and Croissant will still be available for download from the dataset page and API. +### Harvesting Handle Missing Controlled Values +Allows datasets to be harvested with Controlled Vocabulary Values that existed in the originating Dataverse Project but are not in the harvesting Dataverse Project. For mor information view the changes to the endpoint [here](#harvesting-client-endpoint-extended). ### Add .QPJ and .QMD Extensions to Shapefile Handling -- Support for **.qpj** and **.qmd** files in shapefile uploads has been introduced, ensuring that these files are properly recognized and handled as part of geospatial datasets in Dataverse. +Support for **.qpj** and **.qmd** files in shapefile uploads has been introduced, ensuring that these files are properly recognized and handled as part of geospatial datasets in Dataverse. ### Ingested Tabular Data Files Can Be Stored Without the Variable Name Header @@ -144,14 +144,19 @@ New Uningest/Reingest options are available in the File Page Edit menu. Ingest e The /api/files//uningest api also now allows users who can publish the dataset to undo an ingest failure. +### Sphinx Guides now Support Markdown Format and Tabs +Our guides now support the Markdown format with the extension **.md**. Additionally, an option to create tabs in the guides using [Sphinx Tabs](https://sphinx-tabs.readthedocs.io) has been added. (You can see the tabs in action in the "dev usage" page of the Container Guide.) To continue building the guides, you will need to install this new dependency by re-running: +``` +pip install -r requirements.txt. +``` ## 🪲 Bug fixes ### Publication Status Facet Restored In version 6.1, the publication status facet location was unintentionally moved to the bottom. In this version, we have restored the original order. -### Permissions Required To Assign a Role Have Been Fixed +### Assign a Role With Higher Permissions Than Its Own Role Has Been Fixed The permissions required to assign a role have been fixed. It is no longer possible to assign a role that includes permissions that the assigning user doesn't have. @@ -267,7 +272,6 @@ Including the parameter and setting it to true will add a hierarchy showing whic The API endpoint `api/datasets/{id}/metadata` has been changed to default to the latest version of the dataset that the user has access. - ## 📖 Guides ### Container Guide, Documentation for Faster Redeploy @@ -276,68 +280,20 @@ In the Container Guide, documentation for developers on how to quickly redeploy Also in the context of containers, a new option to skip deployment has been added and the war file is now consistently named "dataverse.war" rather than having a version in the filename, such as "dataverse-6.1.war". This predictability makes tooling easier. -Finally, an option to create tabs in the guides using [Sphinx Tabs](https://sphinx-tabs.readthedocs.io) has been added. (You can see the tabs in action in the "dev usage" page of the Container Guide.) To continue building the guides, you will need to install this new dependency by re-running `pip install -r requirements.txt`. - ### Evaluation Version Tutorial on the Containers Guide -The Container Guide now containers a tutorial for running Dataverse in containers for demo or evaluation purposes: https://guides.dataverse.org/en/6.2/container +The Container Guide now containers a tutorial for running Dataverse in containers for demo or evaluation purposes: https://guides.dataverse.org/en/container ### New QA Guide -A new QA Guide is intended mostly for the core development team but may be of interest to contributors. +A new QA Guide is intended mostly for the core development team but may be of interest to contributors on: https://guides.dataverse.org/en/develop/qa -*** +## ⚠️ Breaking Changes https://guides.dataverse.org/en/en/develop/qa/index.html -# ⚙️ New Settings +To view a list of changes that can be impactful to your implementation please visit our detailed [list of changes to the API](https://guides.dataverse.org/en/develop/api/changelog.html). - -### Microprofile settings : - -*The * indicates a provider id indicating which provider the setting is for* - -> - dataverse.pid.providers -> - dataverse.pid.default-provider -> - dataverse.pid.*.type -> - dataverse.pid.*.label -> - dataverse.pid.*.authority -> - dataverse.pid.*.shoulder -> - dataverse.pid.*.identifier-generation-style -> - dataverse.pid.*.datafile-pid-format -> - dataverse.pid.*.managed-list -> - dataverse.pid.*.excluded-list -> - dataverse.pid.*.datacite.mds-api-url -> - dataverse.pid.*.datacite.rest-api-url -> - dataverse.pid.*.datacite.username -> - dataverse.pid.*.datacite.password -> - dataverse.pid.*.ezid.api-url -> - dataverse.pid.*.ezid.username -> - dataverse.pid.*.ezid.password -> - dataverse.pid.*.permalink.base-url -> - dataverse.pid.*.permalink.separator -> - dataverse.pid.*.handlenet.index -> - dataverse.pid.*.handlenet.independent-service -> - dataverse.pid.*.handlenet.auth-handle -> - dataverse.pid.*.handlenet.key.path -> - dataverse.pid.*.handlenet.key.passphrase -> - dataverse.spi.pidproviders.directory - -[⬅️ Go back](#multiple-pid-sup) - -## SMTP Settings: - -> - dataverse.mail.system-email -> - dataverse.mail.mta.host -> - dataverse.mail.mta.port -> - dataverse.mail.mta.ssl.enable -> - dataverse.mail.mta.auth -> - dataverse.mail.mta.user -> - dataverse.mail.mta.password -> - dataverse.mail.mta.allow-utf8-addresses -> - Plus many more for advanced usage and special provider requirements. See [configuration guide for a full list](https://guides.dataverse.org/en/6.2/installation/config.html#dataverse-mail-mta). -[⬅️ Go back](#simplified-smtp-configuration) - -## Upgrade instructions +## 💻 Upgrade instructions Upgrading requires a maintenance window and downtime. Please plan ahead, create backups of your database, etc. These instructions assume that you've already upgraded through all the 5.x releases and are now running Dataverse 6.1. @@ -437,4 +393,54 @@ OR, alternatively, you can edit the following line in your schema.xml by hand as - Restart Solr instance (usually `service solr restart` depending on solr/OS) -8\. Run ReExportAll to update dataset metadata exports. Follow the directions in the [Admin Guide](http://guides.dataverse.org/en/5.14/admin/metadataexport.html#batch-exports-through-the-api). \ No newline at end of file +8\. Run ReExportAll to update dataset metadata exports. Follow the directions in the [Admin Guide](http://guides.dataverse.org/en/5.14/admin/metadataexport.html#batch-exports-through-the-api). + +*** + +## ⚙️ New Settings + +### Microprofile settings + +*The * indicates a provider id indicating which provider the setting is for* + +> - dataverse.pid.providers +> - dataverse.pid.default-provider +> - dataverse.pid.*.type +> - dataverse.pid.*.label +> - dataverse.pid.*.authority +> - dataverse.pid.*.shoulder +> - dataverse.pid.*.identifier-generation-style +> - dataverse.pid.*.datafile-pid-format +> - dataverse.pid.*.managed-list +> - dataverse.pid.*.excluded-list +> - dataverse.pid.*.datacite.mds-api-url +> - dataverse.pid.*.datacite.rest-api-url +> - dataverse.pid.*.datacite.username +> - dataverse.pid.*.datacite.password +> - dataverse.pid.*.ezid.api-url +> - dataverse.pid.*.ezid.username +> - dataverse.pid.*.ezid.password +> - dataverse.pid.*.permalink.base-url +> - dataverse.pid.*.permalink.separator +> - dataverse.pid.*.handlenet.index +> - dataverse.pid.*.handlenet.independent-service +> - dataverse.pid.*.handlenet.auth-handle +> - dataverse.pid.*.handlenet.key.path +> - dataverse.pid.*.handlenet.key.passphrase +> - dataverse.spi.pidproviders.directory + +[⬅️ Go back](#multiple-pid-sup) + +## SMTP Settings: + +> - dataverse.mail.system-email +> - dataverse.mail.mta.host +> - dataverse.mail.mta.port +> - dataverse.mail.mta.ssl.enable +> - dataverse.mail.mta.auth +> - dataverse.mail.mta.user +> - dataverse.mail.mta.password +> - dataverse.mail.mta.allow-utf8-addresses +> - Plus many more for advanced usage and special provider requirements. See [configuration guide for a full list](https://guides.dataverse.org/en/6.2/installation/config.html#dataverse-mail-mta). + +[⬅️ Go back](#simplified-smtp-configuration) \ No newline at end of file From 00c9807840c76b31879f0d6f725d931dbbdd0fee Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva Date: Wed, 27 Mar 2024 14:41:54 -0400 Subject: [PATCH 12/61] Link to the guide code removed --- doc/release-notes/6.2-release-notes.md | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index 8e5bf32b0e1..c1f23c1e703 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -51,6 +51,8 @@ If neither setting exists rate limiting is disabled. In the following example, the default for tier `0` (guest users) is set to 10,000 calls per command per hour and tier `1` (authenticated users) is set to 20,000 calls per command per hour. Tiers not specified in this setting will default to `-1` (No Limit). I.e., -d "10000" is equivalent to -d "10000,-1,-1,..." +For more details check the detailed guide on [this link](https://guides.dataverse.org/en/6.2/installation/config.html#configure-your-dataverse-installation-to-use-jcache-with-hazelcast-as-provided-by-payara-for-rate-limiting). + ``` curl http://localhost:8080/api/admin/settings/:RateLimitingCapacityByTierAndAction -X PUT -d '[{ "tier": 0, @@ -104,7 +106,7 @@ A future major release of Dataverse may remove this way of configuration. Please do take the opportunity to update your SMTP configuration. Details can be found in section of the Installation Guide starting with the [SMTP/Email Configuration](https://guides.dataverse.org/en/6.2/installation/config.html#smtp-email-configuration) section of the Installation Guide. -Once reconfiguration is complete, you should remove legacy, unused config. First, run `asadmin delete-javamail-resource mail/notifyMailSession` as described in the [6.1 guides](https://guides.dataverse.org/en/6.1/installation/installation-main.html#mail-host-configuration-authentication). Then run `curl -X DELETE http://localhost:8080/api/admin/settings/:SystemEmail` as this database setting has been replace with `dataverse.mail.system-email` as described below. +Once reconfiguration is complete, you should remove legacy, unused config. First, run `asadmin delete-javamail-resource mail/notifyMailSession` as described in the [6.1 guides](https://guides.dataverse.org/en/6.2/installation/installation-main.html#mail-host-configuration-authentication). Then run `curl -X DELETE http://localhost:8080/api/admin/settings/:SystemEmail` as this database setting has been replace with `dataverse.mail.system-email` as described below. Please note: as there have been problems with email delivered to SPAM folders when the "From" within mail envelope and the mail session configuration didn't match (#4210), as of this version the sole source for the "From" address is the setting `dataverse.mail.system-email` once you migrate to the new way of configuration. @@ -282,15 +284,15 @@ Also in the context of containers, a new option to skip deployment has been adde ### Evaluation Version Tutorial on the Containers Guide -The Container Guide now containers a tutorial for running Dataverse in containers for demo or evaluation purposes: https://guides.dataverse.org/en/container +The Container Guide now containers a tutorial for running Dataverse in containers for demo or evaluation purposes: https://guides.dataverse.org/en/6.2/container ### New QA Guide -A new QA Guide is intended mostly for the core development team but may be of interest to contributors on: https://guides.dataverse.org/en/develop/qa +A new QA Guide is intended mostly for the core development team but may be of interest to contributors on: https://guides.dataverse.org/en/6.2/develop/qa ## ⚠️ Breaking Changes https://guides.dataverse.org/en/en/develop/qa/index.html -To view a list of changes that can be impactful to your implementation please visit our detailed [list of changes to the API](https://guides.dataverse.org/en/develop/api/changelog.html). +To view a list of changes that can be impactful to your implementation please visit our detailed [list of changes to the API](https://guides.dataverse.org/en/6.2/develop/api/changelog.html). ## 💻 Upgrade instructions @@ -368,7 +370,7 @@ curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/ta 7a\. For installations without custom or experimental metadata blocks: -- Stop Solr instance (usually `service solr stop`, depending on Solr installation/OS, see the [Installation Guide](https://guides.dataverse.org/en/5.14/installation/prerequisites.html#solr-init-script)) +- Stop Solr instance (usually `service solr stop`, depending on Solr installation/OS, see the [Installation Guide](https://guides.dataverse.org/en/6.2/installation/prerequisites.html#solr-init-script)) - Replace schema.xml @@ -378,7 +380,7 @@ curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/ta 7b\. For installations with custom or experimental metadata blocks: -- Stop Solr instance (usually `service solr stop`, depending on Solr installation/OS, see the [Installation Guide](https://guides.dataverse.org/en/5.14/installation/prerequisites.html#solr-init-script)) +- Stop Solr instance (usually `service solr stop`, depending on Solr installation/OS, see the [Installation Guide](https://guides.dataverse.org/en/6.2/installation/prerequisites.html#solr-init-script)) - There are 2 ways to regenerate the schema: Either by collecting the output of the Dataverse schema API and feeding it to the `update-fields.sh` script that we supply, as in the example below (modify the command lines as needed): ``` @@ -393,7 +395,7 @@ OR, alternatively, you can edit the following line in your schema.xml by hand as - Restart Solr instance (usually `service solr restart` depending on solr/OS) -8\. Run ReExportAll to update dataset metadata exports. Follow the directions in the [Admin Guide](http://guides.dataverse.org/en/5.14/admin/metadataexport.html#batch-exports-through-the-api). +8\. Run ReExportAll to update dataset metadata exports. Follow the directions in the [Admin Guide](http://guides.dataverse.org/en/6.2/admin/metadataexport.html#batch-exports-through-the-api). *** @@ -428,6 +430,7 @@ OR, alternatively, you can edit the following line in your schema.xml by hand as > - dataverse.pid.*.handlenet.key.path > - dataverse.pid.*.handlenet.key.passphrase > - dataverse.spi.pidproviders.directory +> - dataverse.solr.concurrency.max-async-indexes [⬅️ Go back](#multiple-pid-sup) From 3556555f718ae64322ac33517130f393090ae357 Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva Date: Wed, 27 Mar 2024 14:54:06 -0400 Subject: [PATCH 13/61] Header placed in the right place --- doc/release-notes/6.2-release-notes.md | 47 ++------------------------ 1 file changed, 3 insertions(+), 44 deletions(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index c1f23c1e703..f5343631b8e 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -5,6 +5,8 @@ Please note: To read these instructions in full, please go to https://github.com This release brings new features, enhancements, and bug fixes to the Dataverse software. Thank you to all of the community members who contributed code, suggestions, bug reports, and other assistance across the project. +## 💡Release Highlights + ### Search and Facet by License A new search facet called "License" has been added and will be displayed as long as there is more than one license in datasets and datafiles in browse/search results. This facet allow you to filter by license such as CC0, etc. @@ -13,8 +15,6 @@ Also, the Search API now handles license filtering using the `fq` parameter, for For more information, see [#10204](https://github.com/IQSS/dataverse/issues/10204). -## 💡Release Highlights - ### Return to Author Now Requires a Reason The Popup for returning to author now allows to type in a message to explain the reasons of return and potential edits needed, that will be sent by email to the author. @@ -53,47 +53,6 @@ Tiers not specified in this setting will default to `-1` (No Limit). I.e., -d "1 For more details check the detailed guide on [this link](https://guides.dataverse.org/en/6.2/installation/config.html#configure-your-dataverse-installation-to-use-jcache-with-hazelcast-as-provided-by-payara-for-rate-limiting). -``` -curl http://localhost:8080/api/admin/settings/:RateLimitingCapacityByTierAndAction -X PUT -d '[{ - "tier": 0, - "limitPerHour": 10, - "actions": [ - "GetLatestPublishedDatasetVersionCommand", - "GetPrivateUrlCommand", - "GetDatasetCommand", - "GetLatestAccessibleDatasetVersionCommand" - ] -}, -{ - "tier": 0, - "limitPerHour": 1, - "actions": [ - "CreateGuestbookResponseCommand", - "UpdateDatasetVersionCommand", - "DestroyDatasetCommand", - "DeleteDataFileCommand", - "FinalizeDatasetPublicationCommand", - "PublishDatasetCommand" - ] -}, -{ - "tier": 1, - "limitPerHour": 30, - "actions": [ - "CreateGuestbookResponseCommand", - "GetLatestPublishedDatasetVersionCommand", - "GetPrivateUrlCommand", - "GetDatasetCommand", - "GetLatestAccessibleDatasetVersionCommand", - "UpdateDatasetVersionCommand", - "DestroyDatasetCommand", - "DeleteDataFileCommand", - "FinalizeDatasetPublicationCommand", - "PublishDatasetCommand" - ] -}]' -``` - Hazelcast is configured in Payara and should not need any changes for this feature ### Simplified SMTP configuration @@ -184,7 +143,7 @@ A bug introduced with the guestboook-at-request, requests are not deleted when g This release adds two missing database constraints that will assure that the externalvocabularyvalue table only has one entry for each uri and that the oaiset table only has one set for each spec. (In the very unlikely case that your existing database has duplicate entries now, install would fail. This can be checked by running ``` -SELECT uri, count(*) FROM externalvocabularyvaluet group by uri; +SELECT uri, count(*) FROM externalvocabularyvalue group by uri; ``` And: ``` From 92c27e7fedfbf4d8918e8f2d8704e2eb20a9ccd0 Mon Sep 17 00:00:00 2001 From: landreev Date: Wed, 27 Mar 2024 15:00:45 -0400 Subject: [PATCH 14/61] Update 6.2-release-notes.md changes to the upgrade instructions order, because of the breaking change in solr schema. --- doc/release-notes/6.2-release-notes.md | 49 +++++++++++++------------- 1 file changed, 25 insertions(+), 24 deletions(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index 8e5bf32b0e1..bc1314d6728 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -308,31 +308,44 @@ In the following commands we assume that Payara 6 is installed in `/usr/local/pa (or `setenv PAYARA /usr/local/payara6` if you are using a `csh`-like shell) -1\. Undeploy the previous version. +1\. Usually, when a Solr schema update is released, we recommend deploying the new version of Dataverse, then updating the `schema.xml` on the solr side. With 6.2, we recommend to install the base schema first. Without it Dataverse 6.2 is not going to be able to show any results after the initial deployment. If your instance is using any custom metadata blocks, you will need to further modify the schema, see the laset step of this instruction (step 8). + +- Stop Solr instance (usually `service solr stop`, depending on Solr installation/OS, see the [Installation Guide](https://guides.dataverse.org/en/5.14/installation/prerequisites.html#solr-init-script)) + +- Replace schema.xml + + - `cd /usr/local/solr/solr-9.3.0/server/solr/collection1/conf` + - `wget https://raw.githubusercontent.com/IQSS/dataverse/master/conf/solr/9.3.0/schema.xml` + +- Start Solr instance (usually `service solr start`, depending on Solr/OS) + +2\. Undeploy the previous version. - `$PAYARA/bin/asadmin undeploy dataverse-6.0` -2\. Stop Payara and remove the generated directory +3\. Stop Payara and remove the generated directory - `service payara stop` - `rm -rf $PAYARA/glassfish/domains/domain1/generated` -3\. Start Payara +4\. Start Payara - `service payara start` -4\. Deploy this version. +5\. Deploy this version. - `$PAYARA/bin/asadmin deploy dataverse-6.1.war` As noted above, deployment of the war file might take several minutes due a database migration script required for the new storage quotas feature. -5\. Restart Payara +6\. Restart Payara - `service payara stop` - `service payara start` -6\. Update Geospatial Metadata Block. +7\. Update the standard Metadata Blocks: + +7a\. Update Geospatial Metadata Block. ``` wget https://github.com/IQSS/dataverse/releases/download/v6.1/geospatial.tsv @@ -340,7 +353,7 @@ As noted above, deployment of the war file might take several minutes due a data curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/tab-separated-values" -X POST --upload-file @geospatial.tsv ``` -6a\. Update Citation Metadata Block. +7b\. Update Citation Metadata Block. ``` wget https://github.com/IQSS/dataverse/releases/download/v6.2/citation.tsv @@ -348,7 +361,7 @@ wget https://github.com/IQSS/dataverse/releases/download/v6.2/citation.tsv curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/tab-separated-values" -X POST --upload-file scripts/api/data/metadatablocks/citation.tsv ``` -6b\. Update Astrophysics Metadata Block. +7c\. Update Astrophysics Metadata Block. ``` wget https://github.com/IQSS/dataverse/releases/download/v6.2/astrophysics.tsv @@ -356,7 +369,7 @@ wget https://github.com/IQSS/dataverse/releases/download/v6.2/astrophysics.tsv curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/tab-separated-values" -X POST --upload-file scripts/api/data/metadatablocks/astrophysics.tsv ``` -6c\. Update Biomedical Metadata Block (to make Alternative Title repeatable) +7d\. Update Biomedical Metadata Block (to make Alternative Title repeatable) ``` wget https://github.com/IQSS/dataverse/releases/download/v6.2/biomedical.tsv @@ -364,19 +377,7 @@ wget https://github.com/IQSS/dataverse/releases/download/v6.2/biomedical.tsv curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/tab-separated-values" -X POST --upload-file scripts/api/data/metadatablocks/biomedical.tsv ``` -7\. Upate Solr schema.xml. See specific instructions below for those installations without custom metadata blocks (7a) and those with custom metadata blocks (7b). - -7a\. For installations without custom or experimental metadata blocks: - -- Stop Solr instance (usually `service solr stop`, depending on Solr installation/OS, see the [Installation Guide](https://guides.dataverse.org/en/5.14/installation/prerequisites.html#solr-init-script)) - -- Replace schema.xml - - - `cp /tmp/dvinstall/schema.xml /usr/local/solr/solr-9.3.0/server/solr/collection1/conf` - -- Start Solr instance (usually `service solr start`, depending on Solr/OS) - -7b\. For installations with custom or experimental metadata blocks: +8\. For installations with custom or experimental metadata blocks: - Stop Solr instance (usually `service solr stop`, depending on Solr installation/OS, see the [Installation Guide](https://guides.dataverse.org/en/5.14/installation/prerequisites.html#solr-init-script)) @@ -393,7 +394,7 @@ OR, alternatively, you can edit the following line in your schema.xml by hand as - Restart Solr instance (usually `service solr restart` depending on solr/OS) -8\. Run ReExportAll to update dataset metadata exports. Follow the directions in the [Admin Guide](http://guides.dataverse.org/en/5.14/admin/metadataexport.html#batch-exports-through-the-api). +9\. Run ReExportAll to update dataset metadata exports. Follow the directions in the [Admin Guide](http://guides.dataverse.org/en/5.14/admin/metadataexport.html#batch-exports-through-the-api). *** @@ -443,4 +444,4 @@ OR, alternatively, you can edit the following line in your schema.xml by hand as > - dataverse.mail.mta.allow-utf8-addresses > - Plus many more for advanced usage and special provider requirements. See [configuration guide for a full list](https://guides.dataverse.org/en/6.2/installation/config.html#dataverse-mail-mta). -[⬅️ Go back](#simplified-smtp-configuration) \ No newline at end of file +[⬅️ Go back](#simplified-smtp-configuration) From aeee15431df48cdb403c84fbc8bea3e2e14ea2c0 Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva Date: Wed, 27 Mar 2024 15:37:41 -0400 Subject: [PATCH 15/61] Rate limit URL change --- doc/release-notes/6.2-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index 83f4572f9a3..54e968684b5 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -51,7 +51,7 @@ If neither setting exists rate limiting is disabled. In the following example, the default for tier `0` (guest users) is set to 10,000 calls per command per hour and tier `1` (authenticated users) is set to 20,000 calls per command per hour. Tiers not specified in this setting will default to `-1` (No Limit). I.e., -d "10000" is equivalent to -d "10000,-1,-1,..." -For more details check the detailed guide on [this link](https://guides.dataverse.org/en/6.2/installation/config.html#configure-your-dataverse-installation-to-use-jcache-with-hazelcast-as-provided-by-payara-for-rate-limiting). +For more details check the detailed guide on [this link](https://guides.dataverse.org/en/6.2/installation/config.html#rate-limiting). Hazelcast is configured in Payara and should not need any changes for this feature From dd0587159cc99f14d53b9dae98a850685d9037d3 Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva Date: Wed, 27 Mar 2024 15:42:50 -0400 Subject: [PATCH 16/61] Collapsed scripts --- doc/release-notes/6.2-release-notes.md | 16 +--------------- 1 file changed, 1 insertion(+), 15 deletions(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index 54e968684b5..cce5db596c7 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -304,35 +304,21 @@ As noted above, deployment of the war file might take several minutes due a data - `service payara stop` - `service payara start` -7\. Update the standard Metadata Blocks: - -7a\. Update Geospatial Metadata Block. +7\. Update the following Metadata Blocks: ``` wget https://github.com/IQSS/dataverse/releases/download/v6.1/geospatial.tsv curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/tab-separated-values" -X POST --upload-file @geospatial.tsv - ``` - -7b\. Update Citation Metadata Block. -``` wget https://github.com/IQSS/dataverse/releases/download/v6.2/citation.tsv curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/tab-separated-values" -X POST --upload-file scripts/api/data/metadatablocks/citation.tsv -``` -7c\. Update Astrophysics Metadata Block. - -``` wget https://github.com/IQSS/dataverse/releases/download/v6.2/astrophysics.tsv curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/tab-separated-values" -X POST --upload-file scripts/api/data/metadatablocks/astrophysics.tsv -``` -7d\. Update Biomedical Metadata Block (to make Alternative Title repeatable) - -``` wget https://github.com/IQSS/dataverse/releases/download/v6.2/biomedical.tsv curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/tab-separated-values" -X POST --upload-file scripts/api/data/metadatablocks/biomedical.tsv From da397eac9c9bb89323928072603f5b7bdb5a3179 Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva Date: Wed, 27 Mar 2024 15:44:32 -0400 Subject: [PATCH 17/61] Incorrect URL on script --- doc/release-notes/6.2-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index cce5db596c7..93a9a6b4960 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -307,7 +307,7 @@ As noted above, deployment of the war file might take several minutes due a data 7\. Update the following Metadata Blocks: ``` - wget https://github.com/IQSS/dataverse/releases/download/v6.1/geospatial.tsv + wget https://github.com/IQSS/dataverse/releases/download/v6.2/geospatial.tsv curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/tab-separated-values" -X POST --upload-file @geospatial.tsv From 5c9cc55643ea6c50184df3c1edb2185a0d97887f Mon Sep 17 00:00:00 2001 From: landreev Date: Wed, 27 Mar 2024 15:50:08 -0400 Subject: [PATCH 18/61] Update 6.2-release-notes.md --- doc/release-notes/6.2-release-notes.md | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index 93a9a6b4960..5350e7f80b7 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -328,16 +328,12 @@ curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/ta - Stop Solr instance (usually `service solr stop`, depending on Solr installation/OS, see the [Installation Guide](https://guides.dataverse.org/en/6.2/installation/prerequisites.html#solr-init-script)) -- There are 2 ways to regenerate the schema: Either by collecting the output of the Dataverse schema API and feeding it to the `update-fields.sh` script that we supply, as in the example below (modify the command lines as needed): +- Run the `update-fields.sh` script that we supply, as in the example below (modify the command lines as needed to reflect the correct path of your solr installation): ``` wget https://raw.githubusercontent.com/IQSS/dataverse/master/conf/solr/9.3.0/update-fields.sh chmod +x update-fields.sh curl "http://localhost:8080/api/admin/index/solr/schema" | ./update-fields.sh /usr/local/solr/solr-9.3.0/server/solr/collection1/conf/schema.xml ``` -OR, alternatively, you can edit the following line in your schema.xml by hand as follows (to indicate that alternative title is now `multiValued="true"`): -``` - -``` - Restart Solr instance (usually `service solr restart` depending on solr/OS) From 51fc6654190cf9639ec8e9824be5925b104daca9 Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva Date: Wed, 27 Mar 2024 16:02:11 -0400 Subject: [PATCH 19/61] Removed the last step reExportAll, it is not required --- doc/release-notes/6.2-release-notes.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index 5350e7f80b7..34c3171c3e1 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -337,8 +337,6 @@ curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/ta - Restart Solr instance (usually `service solr restart` depending on solr/OS) -9\. Run ReExportAll to update dataset metadata exports. Follow the directions in the [Admin Guide](http://guides.dataverse.org/en/5.14/admin/metadataexport.html#batch-exports-through-the-api). - *** ## ⚙️ New Settings From 18ac44fc5ea31010fc0e0583b3d3a9d8f3af3c32 Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva Date: Wed, 27 Mar 2024 16:22:12 -0400 Subject: [PATCH 20/61] Reindex solr change --- doc/release-notes/6.2-release-notes.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index 34c3171c3e1..4017ae37b99 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -337,6 +337,14 @@ curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/ta - Restart Solr instance (usually `service solr restart` depending on solr/OS) +9\. Reindex Solr: + + For details, see https://guides.dataverse.org/en/6.0/admin/solr-search-index.html but here is the reindex command: + +``` + curl http://localhost:8080/api/admin/index +``` + *** ## ⚙️ New Settings From dff1669c82a92a7d75ce26e623bba607b299e72c Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva Date: Wed, 27 Mar 2024 16:32:32 -0400 Subject: [PATCH 21/61] 10424 added --- doc/release-notes/10424-new-api-for-mdc.md | 5 ----- doc/release-notes/6.2-release-notes.md | 4 ++++ 2 files changed, 4 insertions(+), 5 deletions(-) delete mode 100644 doc/release-notes/10424-new-api-for-mdc.md diff --git a/doc/release-notes/10424-new-api-for-mdc.md b/doc/release-notes/10424-new-api-for-mdc.md deleted file mode 100644 index fef8ee2af22..00000000000 --- a/doc/release-notes/10424-new-api-for-mdc.md +++ /dev/null @@ -1,5 +0,0 @@ -(Please put at the bottom of the list under 🌐 API) - -### Experimental Make Data Count processingState API - -An experimental Make Data Count processingState API has been added. For now it has been documented in the developer guide: https://guides.dataverse.org/en/6.2/developers/make-data-count.html#processing-archived-logs diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index 4017ae37b99..854bb606349 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -233,6 +233,10 @@ Including the parameter and setting it to true will add a hierarchy showing whic The API endpoint `api/datasets/{id}/metadata` has been changed to default to the latest version of the dataset that the user has access. +### Experimental Make Data Count processingState API + +An experimental Make Data Count processingState API has been added. For now it has been documented in the (developer guide)[https://guides.dataverse.org/en/6.2/developers/make-data-count.html#processing-archived-logs]. + ## 📖 Guides ### Container Guide, Documentation for Faster Redeploy From bfee2f5e82967fc93152392b6436a806101fc26f Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva Date: Wed, 27 Mar 2024 17:25:41 -0400 Subject: [PATCH 22/61] Fixed pip script --- doc/release-notes/6.2-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index 854bb606349..fe032d74236 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -109,7 +109,7 @@ The /api/files//uningest api also now allows users who can publish the datas Our guides now support the Markdown format with the extension **.md**. Additionally, an option to create tabs in the guides using [Sphinx Tabs](https://sphinx-tabs.readthedocs.io) has been added. (You can see the tabs in action in the "dev usage" page of the Container Guide.) To continue building the guides, you will need to install this new dependency by re-running: ``` -pip install -r requirements.txt. +pip install -r requirements.txt ``` ## 🪲 Bug fixes From 56b2cd51e2af073ad8b478fc6a9bf256854c37db Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva <142103991+jp-tosca@users.noreply.github.com> Date: Thu, 28 Mar 2024 00:36:09 -0400 Subject: [PATCH 23/61] Update 6.2-release-notes.md Co-authored-by: Philip Durbin --- doc/release-notes/6.2-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index fe032d74236..a4a8494a49b 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -15,7 +15,7 @@ Also, the Search API now handles license filtering using the `fq` parameter, for For more information, see [#10204](https://github.com/IQSS/dataverse/issues/10204). -### Return to Author Now Requires a Reason +### When Returning Datasets to Authors, Reviewers Can Add a Note to the Author The Popup for returning to author now allows to type in a message to explain the reasons of return and potential edits needed, that will be sent by email to the author. From 5f5b0c07817dd936c2e998910082fe33759cdc75 Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva <142103991+jp-tosca@users.noreply.github.com> Date: Thu, 28 Mar 2024 00:36:22 -0400 Subject: [PATCH 24/61] Update 6.2-release-notes.md Co-authored-by: Philip Durbin --- doc/release-notes/6.2-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index a4a8494a49b..3f5f80df52f 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -25,7 +25,7 @@ Please note that this note is mandatory, but that you can still type a creative ### Support for Using Multiple PID Providers This release adds support for using multiple PID (DOI, Handle, PermaLink) providers, multiple PID provider accounts -(managing a given protocol, authority,separator, shoulder combination), assigning PID provider accounts to specific collections, +(managing a given protocol, authority, separator, shoulder combination), assigning PID provider accounts to specific collections, and supporting transferred PIDs (where a PID is managed by an account when its authority, separator, and/or shoulder don't match the combination where the account can mint new PIDs). It also adds the ability for additional provider services beyond the existing DataCite, EZId, Handle, and PermaLink providers to be dynamically added as separate jar files. From a67b7f13a996857d1a0b4f30adda269a19e45599 Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva <142103991+jp-tosca@users.noreply.github.com> Date: Thu, 28 Mar 2024 00:37:39 -0400 Subject: [PATCH 25/61] Update 6.2-release-notes.md Co-authored-by: Philip Durbin --- doc/release-notes/6.2-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index 3f5f80df52f..0437940ba53 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -37,7 +37,7 @@ and will be required in a future version. [New microprofile settings](#microprofile-settings) -### Rate Limiting Using JCache (With Hazelcast As Provided by Payara) +### Rate Limiting The option to rate limit has been added to prevent users from over taxing the system either deliberately or by runaway automated processes. Rate limiting can be configured on a tier level with tier 0 being reserved for guest users and tiers 1-any for authenticated users. From ad407c77da5ec2aacb1747c096ae81ddfdd27818 Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva <142103991+jp-tosca@users.noreply.github.com> Date: Thu, 28 Mar 2024 00:38:02 -0400 Subject: [PATCH 26/61] Update 6.2-release-notes.md Co-authored-by: Philip Durbin --- doc/release-notes/6.2-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index 0437940ba53..7a6859698b9 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -55,7 +55,7 @@ For more details check the detailed guide on [this link](https://guides.datavers Hazelcast is configured in Payara and should not need any changes for this feature -### Simplified SMTP configuration +### Simplified SMTP Configuration With this release, we deprecate the usage of `asadmin create-javamail-resource` to configure Dataverse to send mail using your SMTP server and provide a simplified, standard alternative using JVM options or MicroProfile Config. From 8f56d98f1bc7b38e69dacaf02ef2022969147c7e Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva <142103991+jp-tosca@users.noreply.github.com> Date: Thu, 28 Mar 2024 00:38:19 -0400 Subject: [PATCH 27/61] Update 6.2-release-notes.md Co-authored-by: Philip Durbin --- doc/release-notes/6.2-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index 7a6859698b9..746c2702b3b 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -91,7 +91,7 @@ Support for **.qpj** and **.qmd** files in shapefile uploads has been introduced ### Ingested Tabular Data Files Can Be Stored Without the Variable Name Header -Tabular Data Ingest can now save the generated archival files with the list of variable names added as the first tab-delimited line. As the most significant effect of this feature. +Tabular Data Ingest can now save the generated archival files with the list of variable names added as the first tab-delimited line. Access API will be able to take advantage of Direct Download for tab. files saved with these headers on S3 - since they no longer have to be generated and added to the streamed content on the fly. From ab3d9d3b0eec575fbef1935a2ec505607d970a5c Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva <142103991+jp-tosca@users.noreply.github.com> Date: Thu, 28 Mar 2024 00:39:21 -0400 Subject: [PATCH 28/61] Update 6.2-release-notes.md Co-authored-by: Philip Durbin --- doc/release-notes/6.2-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index 746c2702b3b..8e04c93d479 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -93,7 +93,7 @@ Support for **.qpj** and **.qmd** files in shapefile uploads has been introduced Tabular Data Ingest can now save the generated archival files with the list of variable names added as the first tab-delimited line. -Access API will be able to take advantage of Direct Download for tab. files saved with these headers on S3 - since they no longer have to be generated and added to the streamed content on the fly. +Access API will be able to take advantage of Direct Download for .tab files saved with these headers on S3 - since they no longer have to be generated and added to the streamed content on the fly. This behavior is controlled by the new setting **:StoreIngestedTabularFilesWithVarHeaders**. It is false by default, preserving the legacy behavior. When enabled, Dataverse will be able to handle both the newly ingested files, and any already-existing legacy files stored without these headers transparently to the user. E.g. the access API will continue delivering tab-delimited files **with** this header line, whether it needs to add it dynamically for the legacy files, or reading complete files directly from storage for the ones stored with it. From a4a9f8293a0b75e929b9443b5f0384c799067558 Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva <142103991+jp-tosca@users.noreply.github.com> Date: Thu, 28 Mar 2024 00:39:40 -0400 Subject: [PATCH 29/61] Update 6.2-release-notes.md Co-authored-by: Philip Durbin --- doc/release-notes/6.2-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index 8e04c93d479..c52ac13e9db 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -105,7 +105,7 @@ New Uningest/Reingest options are available in the File Page Edit menu. Ingest e The /api/files//uningest api also now allows users who can publish the dataset to undo an ingest failure. -### Sphinx Guides now Support Markdown Format and Tabs +### Sphinx Guides Now Support Markdown Format and Tabs Our guides now support the Markdown format with the extension **.md**. Additionally, an option to create tabs in the guides using [Sphinx Tabs](https://sphinx-tabs.readthedocs.io) has been added. (You can see the tabs in action in the "dev usage" page of the Container Guide.) To continue building the guides, you will need to install this new dependency by re-running: ``` From 917534b44ddbe29ae68e02ea4136f76bb3894db6 Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva <142103991+jp-tosca@users.noreply.github.com> Date: Thu, 28 Mar 2024 00:39:56 -0400 Subject: [PATCH 30/61] Update 6.2-release-notes.md Co-authored-by: Philip Durbin --- doc/release-notes/6.2-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index c52ac13e9db..fe633589115 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -134,7 +134,7 @@ OAI-PMH error handling has been improved to display a machine-readable error in ### Granting File Access Without Access Request -A bug introduced with the guestboook-at-request, requests are not deleted when granted, they are now given the state granted. +A bug introduced with the guestbook-at-request, requests are not deleted when granted, they are now given the state granted. ## 💾 Persistence From 4767a35dd90a9026fa45ee3e4a4e4b7be96e59ce Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva <142103991+jp-tosca@users.noreply.github.com> Date: Thu, 28 Mar 2024 00:41:11 -0400 Subject: [PATCH 31/61] Update 6.2-release-notes.md Co-authored-by: Philip Durbin --- doc/release-notes/6.2-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index fe633589115..20f405829a8 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -187,7 +187,7 @@ This API endpoint indicates if the calling user can download at least one file f The API endpoint `api/harvest/clients/{harvestingClientNickname}` has been extended to include the following fields: -- **allowHarvestingMissingCVV**: enable/disable allowing datasets to be harvested with Controlled Vocabulary Values that existed in the originating Dataverse Project but are not in the harvesting Dataverse Project. Default is false. +- **allowHarvestingMissingCVV**: enable/disable allowing datasets to be harvested with controlled vocabulary values that exist in the originating Dataverse server but are not present in the harvesting Dataverse server. The default is false. *Note: This setting is only available to the API and not currently accessible/settable via the UI* From ef9d3876290b72edc0fee0c69c8e4a8747ebe215 Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva <142103991+jp-tosca@users.noreply.github.com> Date: Thu, 28 Mar 2024 00:41:23 -0400 Subject: [PATCH 32/61] Update 6.2-release-notes.md Co-authored-by: Philip Durbin --- doc/release-notes/6.2-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index 20f405829a8..09d6f70c9f2 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -189,7 +189,7 @@ The API endpoint `api/harvest/clients/{harvestingClientNickname}` has been exten - **allowHarvestingMissingCVV**: enable/disable allowing datasets to be harvested with controlled vocabulary values that exist in the originating Dataverse server but are not present in the harvesting Dataverse server. The default is false. -*Note: This setting is only available to the API and not currently accessible/settable via the UI* +*Note: This setting is only available to the API and not currently accessible/settable via the UI.* ### Version Files Endpoint Extended From 8a03f69447407a1304348d77d4bcfba1f4677caf Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva <142103991+jp-tosca@users.noreply.github.com> Date: Thu, 28 Mar 2024 00:41:30 -0400 Subject: [PATCH 33/61] Update 6.2-release-notes.md Co-authored-by: Philip Durbin --- doc/release-notes/6.2-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index 09d6f70c9f2..3a9f1392735 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -204,7 +204,7 @@ The API endpoint `/api/metadatablocks/{block_id}` has been extended to include t - **displayOrder**: The display order of the field in create/edit forms - **typeClass**: The type class of this field ("controlledVocabulary", "compound", or "primitive") -### Get File Citation As JSON +### Get File Citation as JSON It is now possible to retrieve via API the file citation as it appears on the file landing page. It is formatted in HTML and encoded in JSON. From 5e35796f1eebe6af08620568c0c6f3364f4be602 Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva <142103991+jp-tosca@users.noreply.github.com> Date: Thu, 28 Mar 2024 00:42:12 -0400 Subject: [PATCH 34/61] Update 6.2-release-notes.md Co-authored-by: Philip Durbin --- doc/release-notes/6.2-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index 3a9f1392735..8efdec3c578 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -275,7 +275,7 @@ In the following commands we assume that Payara 6 is installed in `/usr/local/pa 1\. Usually, when a Solr schema update is released, we recommend deploying the new version of Dataverse, then updating the `schema.xml` on the solr side. With 6.2, we recommend to install the base schema first. Without it Dataverse 6.2 is not going to be able to show any results after the initial deployment. If your instance is using any custom metadata blocks, you will need to further modify the schema, see the laset step of this instruction (step 8). -- Stop Solr instance (usually `service solr stop`, depending on Solr installation/OS, see the [Installation Guide](https://guides.dataverse.org/en/5.14/installation/prerequisites.html#solr-init-script)) +- Stop Solr instance (usually `service solr stop`, depending on Solr installation/OS, see the [Installation Guide](https://guides.dataverse.org/en/6.2/installation/prerequisites.html#solr-init-script)) - Replace schema.xml From 8c591c919a5c43f79250a445e02ea591c5a98fcb Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva <142103991+jp-tosca@users.noreply.github.com> Date: Thu, 28 Mar 2024 00:42:37 -0400 Subject: [PATCH 35/61] Update 6.2-release-notes.md Co-authored-by: Philip Durbin --- doc/release-notes/6.2-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index 8efdec3c578..33db417c1dd 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -286,7 +286,7 @@ In the following commands we assume that Payara 6 is installed in `/usr/local/pa 2\. Undeploy the previous version. -- `$PAYARA/bin/asadmin undeploy dataverse-6.0` +- `$PAYARA/bin/asadmin undeploy dataverse-6.1` 3\. Stop Payara and remove the generated directory From 1d04d024472fff294533e184894499f71dcc4d95 Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva <142103991+jp-tosca@users.noreply.github.com> Date: Thu, 28 Mar 2024 00:42:46 -0400 Subject: [PATCH 36/61] Update 6.2-release-notes.md Co-authored-by: Philip Durbin --- doc/release-notes/6.2-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index 33db417c1dd..914856e4d60 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -299,7 +299,7 @@ In the following commands we assume that Payara 6 is installed in `/usr/local/pa 5\. Deploy this version. -- `$PAYARA/bin/asadmin deploy dataverse-6.1.war` +- `$PAYARA/bin/asadmin deploy dataverse-6.2.war` As noted above, deployment of the war file might take several minutes due a database migration script required for the new storage quotas feature. From 40b2152999695ed2a75064c7f9cf0afb39a0697c Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva <142103991+jp-tosca@users.noreply.github.com> Date: Thu, 28 Mar 2024 00:43:00 -0400 Subject: [PATCH 37/61] Update 6.2-release-notes.md Co-authored-by: Philip Durbin --- doc/release-notes/6.2-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index 914856e4d60..a08a14f9403 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -343,7 +343,7 @@ curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/ta 9\. Reindex Solr: - For details, see https://guides.dataverse.org/en/6.0/admin/solr-search-index.html but here is the reindex command: + For details, see https://guides.dataverse.org/en/6.2/admin/solr-search-index.html but here is the reindex command: ``` curl http://localhost:8080/api/admin/index From 25ba92622fdfc8ac8d707c3924c5be0c3180c6af Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva <142103991+jp-tosca@users.noreply.github.com> Date: Thu, 28 Mar 2024 00:43:08 -0400 Subject: [PATCH 38/61] Update 6.2-release-notes.md Co-authored-by: Philip Durbin --- doc/release-notes/6.2-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index a08a14f9403..1de95ecf466 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -353,7 +353,7 @@ curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/ta ## ⚙️ New Settings -### Microprofile settings +### MicroProfile Settings *The * indicates a provider id indicating which provider the setting is for* From 822a1447de5cdb5d448648a49419de9bed38e6bf Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva <142103991+jp-tosca@users.noreply.github.com> Date: Thu, 28 Mar 2024 12:19:06 -0400 Subject: [PATCH 39/61] Update doc/release-notes/6.2-release-notes.md Co-authored-by: Philip Durbin --- doc/release-notes/6.2-release-notes.md | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index 1de95ecf466..f2ef4e84074 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -47,13 +47,7 @@ Two database settings configure the rate limiting. Note: If either of these settings exist in the database rate limiting will be enabled. If neither setting exists rate limiting is disabled. -`:RateLimitingDefaultCapacityTiers` is a comma separated list of default values for each tier. -In the following example, the default for tier `0` (guest users) is set to 10,000 calls per command per hour and tier `1` (authenticated users) is set to 20,000 calls per command per hour. -Tiers not specified in this setting will default to `-1` (No Limit). I.e., -d "10000" is equivalent to -d "10000,-1,-1,..." - -For more details check the detailed guide on [this link](https://guides.dataverse.org/en/6.2/installation/config.html#rate-limiting). - -Hazelcast is configured in Payara and should not need any changes for this feature +For more information check the detailed guide at [this link](https://guides.dataverse.org/en/6.2/installation/config.html#rate-limiting). ### Simplified SMTP Configuration From 4e3a024b13fe9d48949e19f1dee1e64e08265116 Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva Date: Thu, 28 Mar 2024 12:21:17 -0400 Subject: [PATCH 40/61] Rate limit --- doc/release-notes/6.2-release-notes.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index f2ef4e84074..3874cbb7b52 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -42,12 +42,11 @@ and will be required in a future version. The option to rate limit has been added to prevent users from over taxing the system either deliberately or by runaway automated processes. Rate limiting can be configured on a tier level with tier 0 being reserved for guest users and tiers 1-any for authenticated users. Superuser accounts are exempt from rate limiting. + Rate limits can be imposed on command APIs by configuring the tier, the command, and the hourly limit in the database. -Two database settings configure the rate limiting. -Note: If either of these settings exist in the database rate limiting will be enabled. -If neither setting exists rate limiting is disabled. +Two database settings configure the rate limiting **:RateLimitingDefaultCapacityTiers** and **RateLimitingCapacityByTierAndAction**, If either of these settings exist in the database rate limiting will be enabled and If neither setting exists rate limiting is disabled. -For more information check the detailed guide at [this link](https://guides.dataverse.org/en/6.2/installation/config.html#rate-limiting). +For more details check the detailed guide on [this link](https://guides.dataverse.org/en/6.2/installation/config.html#rate-limiting). ### Simplified SMTP Configuration From 4a9c9337cc0ada383a143109becbf45d72a98f52 Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva Date: Thu, 28 Mar 2024 13:12:46 -0400 Subject: [PATCH 41/61] Update with morning chages --- doc/release-notes/6.2-release-notes.md | 42 ++++++++++++++++++++------ 1 file changed, 32 insertions(+), 10 deletions(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index 3874cbb7b52..9bec047cbe3 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -5,15 +5,31 @@ Please note: To read these instructions in full, please go to https://github.com This release brings new features, enhancements, and bug fixes to the Dataverse software. Thank you to all of the community members who contributed code, suggestions, bug reports, and other assistance across the project. +# Index +- [💡Release Highlights](#release-highlights) +- [🪲 Bug fixes](#-bug-fixes) +- [💾 Persistence](#-persistence) +- [🌐 API](#-api) +- [📖 Guides](#-guides) +- [⚠️ Breaking Changes](#-breaking-changes) +- [💻 Upgrade instructions](#-upgrade-instructions) +- [⚙️ New Settings](#-new-settings) + + + ## 💡Release Highlights -### Search and Facet by License +### Search and Facet by License +License have been added to the search facets in the search side panel to filter datasets by license (e.g. CC0). + +Datasets with Custom Terms are aggregated under the "Custom Terms" value of this facet. See the [Licensing](https://guides.dataverse.org/en/6.2/installation/advanced.html#licensing) section of the guide for more details on configured Licenses and Custom Terms. -A new search facet called "License" has been added and will be displayed as long as there is more than one license in datasets and datafiles in browse/search results. This facet allow you to filter by license such as CC0, etc. +For more information, see [#9060](https://github.com/IQSS/dataverse/issues/9060). -Also, the Search API now handles license filtering using the `fq` parameter, for example : `/api/search?q=*&fq=license%3A%22CC0+1.0%22` for CC0 1.0. -For more information, see [#10204](https://github.com/IQSS/dataverse/issues/10204). +Licenses can also be used to filter the Search API results using the `fq` parameter, for example : `/api/search?q=*&fq=license%3A%22CC0+1.0%22` for CC0 1.0, see the [Search API guide](https://guides.dataverse.org/en/6.1/api/search.html) for more examples. + +For more information, see [#10204](https://github.com/IQSS/dataverse/pull/10204). ### When Returning Datasets to Authors, Reviewers Can Add a Note to the Author @@ -34,6 +50,8 @@ These changes require per-provider settings rather than the global PID settings for installations using a single PID Provider account is provided, updating to use the new microprofile settings is highly recommended and will be required in a future version. +For more information check the PID settings on [this link](https://guides.dataverse.org/en/6.2/installation/config.html#global-settings). + [New microprofile settings](#microprofile-settings) @@ -116,7 +134,7 @@ The permissions required to assign a role have been fixed. It is no longer possi ### Geospatial Metadata Block Fields for North and South Renamed -The Geospatial metadata block fields for north and south were labeled incorrectly as ‘Longitudes,’ as reported on #5645. After updating to this version of Dataverse, users will need to update all the endpoints that used ‘northLongitude’ and ‘southLongitude’ to ‘northLatitude’ and ‘southLatitude,’ respectively. +The Geospatial metadata block fields for north and south were labeled incorrectly as ‘Longitudes,’ as reported in #5645. After updating to this version of Dataverse, users will need to update all the endpoints that used ‘northLongitude’ and ‘southLongitude’ to ‘northLatitude’ and ‘southLatitude,’ respectively. ### OAI-PMH Error Handling Has Been Improved @@ -133,7 +151,7 @@ A bug introduced with the guestbook-at-request, requests are not deleted when gr ### Missing Database Constraints -This release adds two missing database constraints that will assure that the externalvocabularyvalue table only has one entry for each uri and that the oaiset table only has one set for each spec. (In the very unlikely case that your existing database has duplicate entries now, install would fail. This can be checked by running +This release adds two missing database constraints that will assure that the externalvocabularyvalue table only has one entry for each uri and that the oaiset table only has one set for each spec. (In the very unlikely case that your existing database has duplicate entries now, install would fail. This can be checked by running the following commands: ``` SELECT uri, count(*) FROM externalvocabularyvalue group by uri; @@ -149,11 +167,11 @@ Then removing any duplicate rows (where count>1). Universe field in variablemetadata table was changed from **varchar(255)** to **text**. The change was made to support longer strings in "universe" metadata field, similar to the rest of text fields in variablemetadata table. -### Postgres Versions +### PostgreSQL Versions -This release adds install script support for the new permissions model in Postgres versions 15+, and bumps FlyWay to support Postgres 16. +This release adds install script support for the new permissions model in PostgreSQL versions 15+, and bumps Flyway to support PostgreSQL 16. -Postgres 13 remains the version used with automated testing. +PostgreSQL 13 remains the version used with automated testing. ## 🌐 API @@ -166,6 +184,8 @@ Listing collection/dataverse role assignments via API still requires ManageDatav A new Index API endpoint has been added allowing an admin to clear an individual dataset from Solr. +For more information visit the documentation on [this link](https://guides.dataverse.org/en/6.2/admin/solr-search-index.html#clearing-a-dataset-from-solr) + ### New Accounts Metrics API Users can retrieve new types of metrics related to user accounts. The new capabilities are [described](https://guides.dataverse.org/en/6.2/api/metrics.html) in the guides. @@ -201,7 +221,9 @@ The API endpoint `/api/metadatablocks/{block_id}` has been extended to include t It is now possible to retrieve via API the file citation as it appears on the file landing page. It is formatted in HTML and encoded in JSON. -This API is not for downloading various citation formats such as EndNote XML, RIS, or BibTeX. This functionality has been requested in [#3140](https://github.com/IQSS/dataverse/issues/3140) and [#9994](https://github.com/IQSS/dataverse/issues/9994) +This API is not for downloading various citation formats such as EndNote XML, RIS, or BibTeX. + +For mor information check the documentation on [this link](https://guides.dataverse.org/en/6.2/api/native-api.html#get-file-citation-as-json) ### Files Endpoint Extended From 4385372235e0ba085995201748ca98ae228d4b2b Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva Date: Thu, 28 Mar 2024 13:14:45 -0400 Subject: [PATCH 42/61] Deleted url --- doc/release-notes/6.2-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index 9bec047cbe3..58cee103f94 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -268,7 +268,7 @@ The Container Guide now containers a tutorial for running Dataverse in container A new QA Guide is intended mostly for the core development team but may be of interest to contributors on: https://guides.dataverse.org/en/6.2/develop/qa -## ⚠️ Breaking Changes https://guides.dataverse.org/en/en/develop/qa/index.html +## ⚠️ Breaking Changes To view a list of changes that can be impactful to your implementation please visit our detailed [list of changes to the API](https://guides.dataverse.org/en/6.2/develop/api/changelog.html). From 452e79c710058e596b89c323e6ff60b35c9e3826 Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva Date: Thu, 28 Mar 2024 13:16:57 -0400 Subject: [PATCH 43/61] Update --- doc/release-notes/6.2-release-notes.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index 58cee103f94..6e162cd6c09 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -11,9 +11,9 @@ Thank you to all of the community members who contributed code, suggestions, bug - [💾 Persistence](#-persistence) - [🌐 API](#-api) - [📖 Guides](#-guides) -- [⚠️ Breaking Changes](#-breaking-changes) +- [⚠️ Breaking Changes](#%EF%B8%8F-breaking-changes) - [💻 Upgrade instructions](#-upgrade-instructions) -- [⚙️ New Settings](#-new-settings) +- [⚙️ New Settings](#%EF%B8%8F-new-settings) From b102bf7b5dbda99f194a8695149dedb0f658a331 Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva Date: Thu, 28 Mar 2024 13:32:25 -0400 Subject: [PATCH 44/61] Latest update post index --- doc/release-notes/6.2-release-notes.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index 6e162cd6c09..bd584feb619 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -242,7 +242,9 @@ The endpoint supports the *includeDeaccessioned* and *returnDatasetVersion* opti The API endpoints for getting datasets, Dataverse collections, and datafiles have been extended to support the following optional 'returnOwners' query parameter. -Including the parameter and setting it to true will add a hierarchy showing which dataset and dataverse collection(s) the object is part of to the json object returned. +Including the parameter and setting it to true will add a hierarchy showing which dataset and dataverse collection(s) the object is part of to the json object returned. + +For more information visit the full native API guide on [this link](https://guides.dataverse.org/en/6.2/api/native-api.html) ### Endpoint Fixed: Datasets Metadata From 9457ea1659872754f47549d6f30ff3049250ca97 Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva Date: Thu, 28 Mar 2024 13:36:22 -0400 Subject: [PATCH 45/61] Change order of index api-breaking --- doc/release-notes/6.2-release-notes.md | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index bd584feb619..335fd606f80 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -10,8 +10,8 @@ Thank you to all of the community members who contributed code, suggestions, bug - [🪲 Bug fixes](#-bug-fixes) - [💾 Persistence](#-persistence) - [🌐 API](#-api) -- [📖 Guides](#-guides) - [⚠️ Breaking Changes](#%EF%B8%8F-breaking-changes) +- [📖 Guides](#-guides) - [💻 Upgrade instructions](#-upgrade-instructions) - [⚙️ New Settings](#%EF%B8%8F-new-settings) @@ -254,6 +254,10 @@ The API endpoint `api/datasets/{id}/metadata` has been changed to default to the An experimental Make Data Count processingState API has been added. For now it has been documented in the (developer guide)[https://guides.dataverse.org/en/6.2/developers/make-data-count.html#processing-archived-logs]. +## ⚠️ Breaking Changes + +To view a list of changes that can be impactful to your implementation please visit our detailed [list of changes to the API](https://guides.dataverse.org/en/6.2/develop/api/changelog.html). + ## 📖 Guides ### Container Guide, Documentation for Faster Redeploy @@ -270,11 +274,6 @@ The Container Guide now containers a tutorial for running Dataverse in container A new QA Guide is intended mostly for the core development team but may be of interest to contributors on: https://guides.dataverse.org/en/6.2/develop/qa -## ⚠️ Breaking Changes - -To view a list of changes that can be impactful to your implementation please visit our detailed [list of changes to the API](https://guides.dataverse.org/en/6.2/develop/api/changelog.html). - - ## 💻 Upgrade instructions Upgrading requires a maintenance window and downtime. Please plan ahead, create backups of your database, etc. From b46c582213e33711e6813108d8b4314e2a04c651 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Thu, 28 Mar 2024 14:11:13 -0400 Subject: [PATCH 46/61] describe #10422 in release notes --- doc/release-notes/10381-index-after-publish.md | 3 --- doc/release-notes/6.2-release-notes.md | 5 +++++ 2 files changed, 5 insertions(+), 3 deletions(-) delete mode 100644 doc/release-notes/10381-index-after-publish.md diff --git a/doc/release-notes/10381-index-after-publish.md b/doc/release-notes/10381-index-after-publish.md deleted file mode 100644 index 84c84d75a28..00000000000 --- a/doc/release-notes/10381-index-after-publish.md +++ /dev/null @@ -1,3 +0,0 @@ -New release adds a new microprofile setting for maximum number of simultaneously running asynchronous dataset index operations that defaults to ``4``: - -dataverse.solr.concurrency.max-async-indexes \ No newline at end of file diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index 335fd606f80..d7eeeebde2f 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -122,6 +122,11 @@ Our guides now support the Markdown format with the extension **.md**. Additiona ``` pip install -r requirements.txt ``` + +### Number of Concurrent Indexing Operations Now Configurable + +A new MicroProfile setting called `dataverse.solr.concurrency.max-async-indexes` has been added that controls the maximum number of simultaneously running asynchronous dataset index operations (defaults to 4). + ## 🪲 Bug fixes ### Publication Status Facet Restored From aac525600b306522d7fa50c1e8201735c922c33e Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Thu, 28 Mar 2024 14:41:07 -0400 Subject: [PATCH 47/61] tweaks #10422 --- doc/release-notes/6.2-release-notes.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index d7eeeebde2f..bd72b78d3d7 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -5,12 +5,12 @@ Please note: To read these instructions in full, please go to https://github.com This release brings new features, enhancements, and bug fixes to the Dataverse software. Thank you to all of the community members who contributed code, suggestions, bug reports, and other assistance across the project. -# Index +# Table of Contents - [💡Release Highlights](#release-highlights) - [🪲 Bug fixes](#-bug-fixes) - [💾 Persistence](#-persistence) - [🌐 API](#-api) -- [⚠️ Breaking Changes](#%EF%B8%8F-breaking-changes) +- [⚠️ Backward Incompatibilities](#%EF%B8%8F-backward-incompatibilities) - [📖 Guides](#-guides) - [💻 Upgrade instructions](#-upgrade-instructions) - [⚙️ New Settings](#%EF%B8%8F-new-settings) @@ -259,7 +259,7 @@ The API endpoint `api/datasets/{id}/metadata` has been changed to default to the An experimental Make Data Count processingState API has been added. For now it has been documented in the (developer guide)[https://guides.dataverse.org/en/6.2/developers/make-data-count.html#processing-archived-logs]. -## ⚠️ Breaking Changes +## ⚠️ Backward Incompatibilities To view a list of changes that can be impactful to your implementation please visit our detailed [list of changes to the API](https://guides.dataverse.org/en/6.2/develop/api/changelog.html). From 8c8fe789ef3f1d000756e6f3017f706ce1d7aedf Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Thu, 28 Mar 2024 14:56:34 -0400 Subject: [PATCH 48/61] add settings, reformat --- doc/release-notes/6.2-release-notes.md | 76 ++++++++++++++------------ 1 file changed, 41 insertions(+), 35 deletions(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index bd72b78d3d7..5b77d4a78c6 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -378,45 +378,51 @@ curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/ta *The * indicates a provider id indicating which provider the setting is for* -> - dataverse.pid.providers -> - dataverse.pid.default-provider -> - dataverse.pid.*.type -> - dataverse.pid.*.label -> - dataverse.pid.*.authority -> - dataverse.pid.*.shoulder -> - dataverse.pid.*.identifier-generation-style -> - dataverse.pid.*.datafile-pid-format -> - dataverse.pid.*.managed-list -> - dataverse.pid.*.excluded-list -> - dataverse.pid.*.datacite.mds-api-url -> - dataverse.pid.*.datacite.rest-api-url -> - dataverse.pid.*.datacite.username -> - dataverse.pid.*.datacite.password -> - dataverse.pid.*.ezid.api-url -> - dataverse.pid.*.ezid.username -> - dataverse.pid.*.ezid.password -> - dataverse.pid.*.permalink.base-url -> - dataverse.pid.*.permalink.separator -> - dataverse.pid.*.handlenet.index -> - dataverse.pid.*.handlenet.independent-service -> - dataverse.pid.*.handlenet.auth-handle -> - dataverse.pid.*.handlenet.key.path -> - dataverse.pid.*.handlenet.key.passphrase -> - dataverse.spi.pidproviders.directory -> - dataverse.solr.concurrency.max-async-indexes +- dataverse.pid.providers +- dataverse.pid.default-provider +- dataverse.pid.*.type +- dataverse.pid.*.label +- dataverse.pid.*.authority +- dataverse.pid.*.shoulder +- dataverse.pid.*.identifier-generation-style +- dataverse.pid.*.datafile-pid-format +- dataverse.pid.*.managed-list +- dataverse.pid.*.excluded-list +- dataverse.pid.*.datacite.mds-api-url +- dataverse.pid.*.datacite.rest-api-url +- dataverse.pid.*.datacite.username +- dataverse.pid.*.datacite.password +- dataverse.pid.*.ezid.api-url +- dataverse.pid.*.ezid.username +- dataverse.pid.*.ezid.password +- dataverse.pid.*.permalink.base-url +- dataverse.pid.*.permalink.separator +- dataverse.pid.*.handlenet.index +- dataverse.pid.*.handlenet.independent-service +- dataverse.pid.*.handlenet.auth-handle +- dataverse.pid.*.handlenet.key.path +- dataverse.pid.*.handlenet.key.passphrase +- dataverse.spi.pidproviders.directory +- dataverse.solr.concurrency.max-async-indexes [⬅️ Go back](#multiple-pid-sup) ## SMTP Settings: -> - dataverse.mail.system-email -> - dataverse.mail.mta.host -> - dataverse.mail.mta.port -> - dataverse.mail.mta.ssl.enable -> - dataverse.mail.mta.auth -> - dataverse.mail.mta.user -> - dataverse.mail.mta.password -> - dataverse.mail.mta.allow-utf8-addresses -> - Plus many more for advanced usage and special provider requirements. See [configuration guide for a full list](https://guides.dataverse.org/en/6.2/installation/config.html#dataverse-mail-mta). +- dataverse.mail.system-email +- dataverse.mail.mta.host +- dataverse.mail.mta.port +- dataverse.mail.mta.ssl.enable +- dataverse.mail.mta.auth +- dataverse.mail.mta.user +- dataverse.mail.mta.password +- dataverse.mail.mta.allow-utf8-addresses +- Plus many more for advanced usage and special provider requirements. See [configuration guide for a full list](https://guides.dataverse.org/en/6.2/installation/config.html#dataverse-mail-mta). [⬅️ Go back](#simplified-smtp-configuration) + +## Database Settings: + +- :RateLimitingDefaultCapacityTiers +- :RateLimitingCapacityByTierAndAction +- :StoreIngestedTabularFilesWithVarHeaders From 950146c23f862007cfc6c3531235888905268781 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Thu, 28 Mar 2024 15:00:52 -0400 Subject: [PATCH 49/61] a few more sections, move settings --- doc/release-notes/6.2-release-notes.md | 119 +++++++++++++------------ 1 file changed, 63 insertions(+), 56 deletions(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index 5b77d4a78c6..81db2e3a93e 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -279,6 +279,69 @@ The Container Guide now containers a tutorial for running Dataverse in container A new QA Guide is intended mostly for the core development team but may be of interest to contributors on: https://guides.dataverse.org/en/6.2/develop/qa +## ⚙️ New Settings + +### MicroProfile Settings + +*The * indicates a provider id indicating which provider the setting is for* + +- dataverse.pid.providers +- dataverse.pid.default-provider +- dataverse.pid.*.type +- dataverse.pid.*.label +- dataverse.pid.*.authority +- dataverse.pid.*.shoulder +- dataverse.pid.*.identifier-generation-style +- dataverse.pid.*.datafile-pid-format +- dataverse.pid.*.managed-list +- dataverse.pid.*.excluded-list +- dataverse.pid.*.datacite.mds-api-url +- dataverse.pid.*.datacite.rest-api-url +- dataverse.pid.*.datacite.username +- dataverse.pid.*.datacite.password +- dataverse.pid.*.ezid.api-url +- dataverse.pid.*.ezid.username +- dataverse.pid.*.ezid.password +- dataverse.pid.*.permalink.base-url +- dataverse.pid.*.permalink.separator +- dataverse.pid.*.handlenet.index +- dataverse.pid.*.handlenet.independent-service +- dataverse.pid.*.handlenet.auth-handle +- dataverse.pid.*.handlenet.key.path +- dataverse.pid.*.handlenet.key.passphrase +- dataverse.spi.pidproviders.directory +- dataverse.solr.concurrency.max-async-indexes + +[⬅️ Go back](#multiple-pid-sup) + +## SMTP Settings: + +- dataverse.mail.system-email +- dataverse.mail.mta.host +- dataverse.mail.mta.port +- dataverse.mail.mta.ssl.enable +- dataverse.mail.mta.auth +- dataverse.mail.mta.user +- dataverse.mail.mta.password +- dataverse.mail.mta.allow-utf8-addresses +- Plus many more for advanced usage and special provider requirements. See [configuration guide for a full list](https://guides.dataverse.org/en/6.2/installation/config.html#dataverse-mail-mta). + +[⬅️ Go back](#simplified-smtp-configuration) + +## Database Settings: + +- :RateLimitingDefaultCapacityTiers +- :RateLimitingCapacityByTierAndAction +- :StoreIngestedTabularFilesWithVarHeaders + +## Complete List of Changes + +For the complete list of code changes in this release, see the [6.2 Milestone](https://github.com/IQSS/dataverse/issues?q=milestone%3A6.2+is%3Aclosed) in GitHub. + +## Getting Help + +For help with upgrading, installing, or general questions please post to the [Dataverse Community Google Group](https://groups.google.com/forum/#!forum/dataverse-community) or email support@dataverse.org. + ## 💻 Upgrade instructions Upgrading requires a maintenance window and downtime. Please plan ahead, create backups of your database, etc. @@ -370,59 +433,3 @@ curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/ta curl http://localhost:8080/api/admin/index ``` -*** - -## ⚙️ New Settings - -### MicroProfile Settings - -*The * indicates a provider id indicating which provider the setting is for* - -- dataverse.pid.providers -- dataverse.pid.default-provider -- dataverse.pid.*.type -- dataverse.pid.*.label -- dataverse.pid.*.authority -- dataverse.pid.*.shoulder -- dataverse.pid.*.identifier-generation-style -- dataverse.pid.*.datafile-pid-format -- dataverse.pid.*.managed-list -- dataverse.pid.*.excluded-list -- dataverse.pid.*.datacite.mds-api-url -- dataverse.pid.*.datacite.rest-api-url -- dataverse.pid.*.datacite.username -- dataverse.pid.*.datacite.password -- dataverse.pid.*.ezid.api-url -- dataverse.pid.*.ezid.username -- dataverse.pid.*.ezid.password -- dataverse.pid.*.permalink.base-url -- dataverse.pid.*.permalink.separator -- dataverse.pid.*.handlenet.index -- dataverse.pid.*.handlenet.independent-service -- dataverse.pid.*.handlenet.auth-handle -- dataverse.pid.*.handlenet.key.path -- dataverse.pid.*.handlenet.key.passphrase -- dataverse.spi.pidproviders.directory -- dataverse.solr.concurrency.max-async-indexes - -[⬅️ Go back](#multiple-pid-sup) - -## SMTP Settings: - -- dataverse.mail.system-email -- dataverse.mail.mta.host -- dataverse.mail.mta.port -- dataverse.mail.mta.ssl.enable -- dataverse.mail.mta.auth -- dataverse.mail.mta.user -- dataverse.mail.mta.password -- dataverse.mail.mta.allow-utf8-addresses -- Plus many more for advanced usage and special provider requirements. See [configuration guide for a full list](https://guides.dataverse.org/en/6.2/installation/config.html#dataverse-mail-mta). - -[⬅️ Go back](#simplified-smtp-configuration) - -## Database Settings: - -- :RateLimitingDefaultCapacityTiers -- :RateLimitingCapacityByTierAndAction -- :StoreIngestedTabularFilesWithVarHeaders From bfd94074018d7dd85b1c5b6a0632ab5ec5311ab8 Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva Date: Thu, 28 Mar 2024 15:12:09 -0400 Subject: [PATCH 50/61] Table of contets reorg --- doc/release-notes/6.2-release-notes.md | 18 +++++++----------- 1 file changed, 7 insertions(+), 11 deletions(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index 81db2e3a93e..6becea8d05d 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -12,10 +12,10 @@ Thank you to all of the community members who contributed code, suggestions, bug - [🌐 API](#-api) - [⚠️ Backward Incompatibilities](#%EF%B8%8F-backward-incompatibilities) - [📖 Guides](#-guides) -- [💻 Upgrade instructions](#-upgrade-instructions) - [⚙️ New Settings](#%EF%B8%8F-new-settings) - - +- [📋 Complete List of Changes](#-upgrade-instructions) +- [🛟 Getting Help](#-upgrade-instructions) +- [💻 Upgrade instructions](#-upgrade-instructions) ## 💡Release Highlights @@ -312,9 +312,7 @@ A new QA Guide is intended mostly for the core development team but may be of in - dataverse.spi.pidproviders.directory - dataverse.solr.concurrency.max-async-indexes -[⬅️ Go back](#multiple-pid-sup) - -## SMTP Settings: +### SMTP Settings: - dataverse.mail.system-email - dataverse.mail.mta.host @@ -326,19 +324,17 @@ A new QA Guide is intended mostly for the core development team but may be of in - dataverse.mail.mta.allow-utf8-addresses - Plus many more for advanced usage and special provider requirements. See [configuration guide for a full list](https://guides.dataverse.org/en/6.2/installation/config.html#dataverse-mail-mta). -[⬅️ Go back](#simplified-smtp-configuration) - -## Database Settings: +### Database Settings: - :RateLimitingDefaultCapacityTiers - :RateLimitingCapacityByTierAndAction - :StoreIngestedTabularFilesWithVarHeaders -## Complete List of Changes +## 📋 Complete List of Changes For the complete list of code changes in this release, see the [6.2 Milestone](https://github.com/IQSS/dataverse/issues?q=milestone%3A6.2+is%3Aclosed) in GitHub. -## Getting Help +## 🛟 Getting Help For help with upgrading, installing, or general questions please post to the [Dataverse Community Google Group](https://groups.google.com/forum/#!forum/dataverse-community) or email support@dataverse.org. From 5f7c99f94ad5f9888dcdccf3fa4a43c80ffe9640 Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva Date: Thu, 28 Mar 2024 15:18:07 -0400 Subject: [PATCH 51/61] Table of contents final --- doc/release-notes/6.2-release-notes.md | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index 6becea8d05d..d172ba48c29 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -13,8 +13,8 @@ Thank you to all of the community members who contributed code, suggestions, bug - [⚠️ Backward Incompatibilities](#%EF%B8%8F-backward-incompatibilities) - [📖 Guides](#-guides) - [⚙️ New Settings](#%EF%B8%8F-new-settings) -- [📋 Complete List of Changes](#-upgrade-instructions) -- [🛟 Getting Help](#-upgrade-instructions) +- [📋 Complete List of Changes](#-complete-list-of-changes) +- [🛟 Getting Help](#-getting-help) - [💻 Upgrade instructions](#-upgrade-instructions) ## 💡Release Highlights @@ -127,6 +127,8 @@ pip install -r requirements.txt A new MicroProfile setting called `dataverse.solr.concurrency.max-async-indexes` has been added that controls the maximum number of simultaneously running asynchronous dataset index operations (defaults to 4). +[⬆️](#-table-of-contents) + ## 🪲 Bug fixes ### Publication Status Facet Restored @@ -152,6 +154,8 @@ OAI-PMH error handling has been improved to display a machine-readable error in A bug introduced with the guestbook-at-request, requests are not deleted when granted, they are now given the state granted. +[⬆️](#-table-of-contents) + ## 💾 Persistence ### Missing Database Constraints @@ -259,10 +263,14 @@ The API endpoint `api/datasets/{id}/metadata` has been changed to default to the An experimental Make Data Count processingState API has been added. For now it has been documented in the (developer guide)[https://guides.dataverse.org/en/6.2/developers/make-data-count.html#processing-archived-logs]. +[⬆️](#-table-of-contents) + ## ⚠️ Backward Incompatibilities To view a list of changes that can be impactful to your implementation please visit our detailed [list of changes to the API](https://guides.dataverse.org/en/6.2/develop/api/changelog.html). +[⬆️](#-table-of-contents) + ## 📖 Guides ### Container Guide, Documentation for Faster Redeploy @@ -279,6 +287,8 @@ The Container Guide now containers a tutorial for running Dataverse in container A new QA Guide is intended mostly for the core development team but may be of interest to contributors on: https://guides.dataverse.org/en/6.2/develop/qa +[⬆️](#-table-of-contents) + ## ⚙️ New Settings ### MicroProfile Settings @@ -334,10 +344,14 @@ A new QA Guide is intended mostly for the core development team but may be of in For the complete list of code changes in this release, see the [6.2 Milestone](https://github.com/IQSS/dataverse/issues?q=milestone%3A6.2+is%3Aclosed) in GitHub. +[⬆️](#-table-of-contents) + ## 🛟 Getting Help For help with upgrading, installing, or general questions please post to the [Dataverse Community Google Group](https://groups.google.com/forum/#!forum/dataverse-community) or email support@dataverse.org. +[⬆️](#-table-of-contents) + ## 💻 Upgrade instructions Upgrading requires a maintenance window and downtime. Please plan ahead, create backups of your database, etc. @@ -428,4 +442,4 @@ curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/ta ``` curl http://localhost:8080/api/admin/index ``` - +[⬆️](#-table-of-contents) From f82ebdaaa27f3e183a5c9f43fa100eacf20e2a9d Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva Date: Thu, 28 Mar 2024 15:20:08 -0400 Subject: [PATCH 52/61] TOC fix --- doc/release-notes/6.2-release-notes.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index d172ba48c29..bbf8d010ebc 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -127,7 +127,7 @@ pip install -r requirements.txt A new MicroProfile setting called `dataverse.solr.concurrency.max-async-indexes` has been added that controls the maximum number of simultaneously running asynchronous dataset index operations (defaults to 4). -[⬆️](#-table-of-contents) +[⬆️](#table-of-contents) ## 🪲 Bug fixes @@ -154,7 +154,7 @@ OAI-PMH error handling has been improved to display a machine-readable error in A bug introduced with the guestbook-at-request, requests are not deleted when granted, they are now given the state granted. -[⬆️](#-table-of-contents) +[⬆️](#table-of-contents) ## 💾 Persistence @@ -263,13 +263,13 @@ The API endpoint `api/datasets/{id}/metadata` has been changed to default to the An experimental Make Data Count processingState API has been added. For now it has been documented in the (developer guide)[https://guides.dataverse.org/en/6.2/developers/make-data-count.html#processing-archived-logs]. -[⬆️](#-table-of-contents) +[⬆️](#table-of-contents) ## ⚠️ Backward Incompatibilities To view a list of changes that can be impactful to your implementation please visit our detailed [list of changes to the API](https://guides.dataverse.org/en/6.2/develop/api/changelog.html). -[⬆️](#-table-of-contents) +[⬆️](#table-of-contents) ## 📖 Guides @@ -287,7 +287,7 @@ The Container Guide now containers a tutorial for running Dataverse in container A new QA Guide is intended mostly for the core development team but may be of interest to contributors on: https://guides.dataverse.org/en/6.2/develop/qa -[⬆️](#-table-of-contents) +[⬆️](#table-of-contents) ## ⚙️ New Settings @@ -344,13 +344,13 @@ A new QA Guide is intended mostly for the core development team but may be of in For the complete list of code changes in this release, see the [6.2 Milestone](https://github.com/IQSS/dataverse/issues?q=milestone%3A6.2+is%3Aclosed) in GitHub. -[⬆️](#-table-of-contents) +[⬆️](#table-of-contents) ## 🛟 Getting Help For help with upgrading, installing, or general questions please post to the [Dataverse Community Google Group](https://groups.google.com/forum/#!forum/dataverse-community) or email support@dataverse.org. -[⬆️](#-table-of-contents) +[⬆️](#table-of-contents) ## 💻 Upgrade instructions Upgrading requires a maintenance window and downtime. Please plan ahead, create backups of your database, etc. @@ -442,4 +442,4 @@ curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/ta ``` curl http://localhost:8080/api/admin/index ``` -[⬆️](#-table-of-contents) +[⬆️](#table-of-contents) From 2035585303925b2cc8ef0fc4d74bc5aee636d67b Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva Date: Thu, 28 Mar 2024 15:22:44 -0400 Subject: [PATCH 53/61] toc fix 2 --- doc/release-notes/6.2-release-notes.md | 1 + 1 file changed, 1 insertion(+) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index bbf8d010ebc..494bab39553 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -182,6 +182,7 @@ This release adds install script support for the new permissions model in Postgr PostgreSQL 13 remains the version used with automated testing. +[⬆️](#table-of-contents) ## 🌐 API From 8bbe9744e8945a3318a61f3a23686950d6ca1536 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Thu, 28 Mar 2024 15:32:31 -0400 Subject: [PATCH 54/61] more links, tweaks --- doc/release-notes/6.2-release-notes.md | 28 +++++++++++++++++++------- 1 file changed, 21 insertions(+), 7 deletions(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index 494bab39553..3c2a116ac7a 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -37,6 +37,8 @@ The Popup for returning to author now allows to type in a message to explain the Please note that this note is mandatory, but that you can still type a creative and meaningful comment such as "The author would like to modify his dataset", "Files are missing", "Nothing to report" or "A curation report with comments and suggestions/instructions will follow in another email" that suits your situation. +For more information, see #10137. + ### Support for Using Multiple PID Providers @@ -92,14 +94,18 @@ For more information, see [#10360](https://github.com/IQSS/dataverse/issues/1036 When a Dataverse installation is configured to use a metadata exporter for the [Croissant](https://github.com/mlcommons/croissant) format, the content of the JSON-LD in the **<head>** of dataset landing pages will be replaced with that format. However, both JSON-LD and Croissant will still be available for download from the dataset page and API. +For more information, see #10382. + ### Harvesting Handle Missing Controlled Values -Allows datasets to be harvested with Controlled Vocabulary Values that existed in the originating Dataverse Project but are not in the harvesting Dataverse Project. For mor information view the changes to the endpoint [here](#harvesting-client-endpoint-extended). +Allows datasets to be harvested with Controlled Vocabulary Values that existed in the originating Dataverse Project but are not in the harvesting Dataverse Project. For more information, view the changes to the endpoint [here](#harvesting-client-endpoint-extended). ### Add .QPJ and .QMD Extensions to Shapefile Handling Support for **.qpj** and **.qmd** files in shapefile uploads has been introduced, ensuring that these files are properly recognized and handled as part of geospatial datasets in Dataverse. +For more information, see #10305. + ### Ingested Tabular Data Files Can Be Stored Without the Variable Name Header Tabular Data Ingest can now save the generated archival files with the list of variable names added as the first tab-delimited line. @@ -110,12 +116,16 @@ This behavior is controlled by the new setting **:StoreIngestedTabularFilesWithV We are planning to add an API for converting existing legacy tabular files in a future release. +For more information, see #10282. + ### Uningest/Reingest Options Available in the File Page Edit Menu New Uningest/Reingest options are available in the File Page Edit menu. Ingest errors can be cleared by users who can published the associated dataset and by superusers, allowing for a successful ingest to be undone or retried (e.g. after a Dataverse version update or if ingest size limits are changed). The /api/files//uningest api also now allows users who can publish the dataset to undo an ingest failure. +For more information, see #10319. + ### Sphinx Guides Now Support Markdown Format and Tabs Our guides now support the Markdown format with the extension **.md**. Additionally, an option to create tabs in the guides using [Sphinx Tabs](https://sphinx-tabs.readthedocs.io) has been added. (You can see the tabs in action in the "dev usage" page of the Container Guide.) To continue building the guides, you will need to install this new dependency by re-running: @@ -123,10 +133,14 @@ Our guides now support the Markdown format with the extension **.md**. Additiona pip install -r requirements.txt ``` +For more information, see #10111. + ### Number of Concurrent Indexing Operations Now Configurable A new MicroProfile setting called `dataverse.solr.concurrency.max-async-indexes` has been added that controls the maximum number of simultaneously running asynchronous dataset index operations (defaults to 4). +For more information, see #10388. + [⬆️](#table-of-contents) ## 🪲 Bug fixes @@ -141,14 +155,14 @@ The permissions required to assign a role have been fixed. It is no longer possi ### Geospatial Metadata Block Fields for North and South Renamed -The Geospatial metadata block fields for north and south were labeled incorrectly as ‘Longitudes,’ as reported in #5645. After updating to this version of Dataverse, users will need to update all the endpoints that used ‘northLongitude’ and ‘southLongitude’ to ‘northLatitude’ and ‘southLatitude,’ respectively. +The Geospatial metadata block fields for north and south were labeled incorrectly as longitudes, as reported in #5645. After updating to this version of Dataverse, users will need to update any API client code used "northLongitude" and "southLongitude" to "northLatitude" and "southLatitude", respectively, as [mentioned](https://groups.google.com/g/dataverse-community/c/5qpOIZUSL6A/m/nlYGEXkYAAAJ) on the mailing list. ### OAI-PMH Error Handling Has Been Improved OAI-PMH error handling has been improved to display a machine-readable error in XML rather than a 500 error with no further information. -> - /oai?foo=bar will show "No argument 'verb' found" -> - /oai?verb=foo&verb=bar will show "Verb must be singular, given: '[foo, bar]'" +- /oai?foo=bar will show "No argument 'verb' found" +- /oai?verb=foo&verb=bar will show "Verb must be singular, given: '[foo, bar]'" ### Granting File Access Without Access Request @@ -233,7 +247,7 @@ It is now possible to retrieve via API the file citation as it appears on the fi This API is not for downloading various citation formats such as EndNote XML, RIS, or BibTeX. -For mor information check the documentation on [this link](https://guides.dataverse.org/en/6.2/api/native-api.html#get-file-citation-as-json) +For more information check the documentation on [this link](https://guides.dataverse.org/en/6.2/api/native-api.html#get-file-citation-as-json) ### Files Endpoint Extended @@ -282,7 +296,7 @@ Also in the context of containers, a new option to skip deployment has been adde ### Evaluation Version Tutorial on the Containers Guide -The Container Guide now containers a tutorial for running Dataverse in containers for demo or evaluation purposes: https://guides.dataverse.org/en/6.2/container +The Container Guide now containers a tutorial for running Dataverse in containers for demo or evaluation purposes: https://guides.dataverse.org/en/6.2/container/running/demo.html ### New QA Guide @@ -353,7 +367,7 @@ For help with upgrading, installing, or general questions please post to the [Da [⬆️](#table-of-contents) -## 💻 Upgrade instructions +## 💻 Upgrade Instructions Upgrading requires a maintenance window and downtime. Please plan ahead, create backups of your database, etc. These instructions assume that you've already upgraded through all the 5.x releases and are now running Dataverse 6.1. From ad91b5b62b8aef2adffad15166a565acac83273e Mon Sep 17 00:00:00 2001 From: Juan Pablo Tosca Villanueva Date: Thu, 28 Mar 2024 16:32:34 -0400 Subject: [PATCH 55/61] Geospatial command fix --- doc/release-notes/6.2-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index 3c2a116ac7a..143215f1bdb 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -422,7 +422,7 @@ As noted above, deployment of the war file might take several minutes due a data ``` wget https://github.com/IQSS/dataverse/releases/download/v6.2/geospatial.tsv - curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/tab-separated-values" -X POST --upload-file @geospatial.tsv + curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/tab-separated-values" -X POST --upload-file scripts/api/data/metadatablocks/geospatial.tsv wget https://github.com/IQSS/dataverse/releases/download/v6.2/citation.tsv From ee14b2686acc886924f3758c583edcbad1929aa5 Mon Sep 17 00:00:00 2001 From: Stephen Kraffmiller Date: Fri, 29 Mar 2024 10:04:52 -0400 Subject: [PATCH 56/61] #10422 incorporate tooltip fix --- doc/release-notes/10397-geospatial-tooltip-fix.md | 2 -- doc/release-notes/6.2-release-notes.md | 1 + 2 files changed, 1 insertion(+), 2 deletions(-) delete mode 100644 doc/release-notes/10397-geospatial-tooltip-fix.md diff --git a/doc/release-notes/10397-geospatial-tooltip-fix.md b/doc/release-notes/10397-geospatial-tooltip-fix.md deleted file mode 100644 index 0774dc1860e..00000000000 --- a/doc/release-notes/10397-geospatial-tooltip-fix.md +++ /dev/null @@ -1,2 +0,0 @@ -We have updated the tooltips in the Geospatial metadata block, where the use of commas instead of dots in coordinate values was incorrectly suggested. - diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index 143215f1bdb..cc4b03bbc20 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -156,6 +156,7 @@ The permissions required to assign a role have been fixed. It is no longer possi ### Geospatial Metadata Block Fields for North and South Renamed The Geospatial metadata block fields for north and south were labeled incorrectly as longitudes, as reported in #5645. After updating to this version of Dataverse, users will need to update any API client code used "northLongitude" and "southLongitude" to "northLatitude" and "southLatitude", respectively, as [mentioned](https://groups.google.com/g/dataverse-community/c/5qpOIZUSL6A/m/nlYGEXkYAAAJ) on the mailing list. +Also, we have updated the tooltips in the Geospatial metadata block, where the use of commas instead of dots in coordinate values was incorrectly suggested. ### OAI-PMH Error Handling Has Been Improved From bd333910adbeccd98d372e19b526674a688ae76d Mon Sep 17 00:00:00 2001 From: Stephen Kraffmiller Date: Fri, 29 Mar 2024 10:13:29 -0400 Subject: [PATCH 57/61] #10442 add note for harvest redirects --- doc/release-notes/10254-fix-harvested-redirects.md | 1 - doc/release-notes/6.2-release-notes.md | 4 ++++ 2 files changed, 4 insertions(+), 1 deletion(-) delete mode 100644 doc/release-notes/10254-fix-harvested-redirects.md diff --git a/doc/release-notes/10254-fix-harvested-redirects.md b/doc/release-notes/10254-fix-harvested-redirects.md deleted file mode 100644 index 02ee5ddaf4d..00000000000 --- a/doc/release-notes/10254-fix-harvested-redirects.md +++ /dev/null @@ -1 +0,0 @@ -Redirects from search cards back to the original source for datasets harvested from "Generic OAI Archives", i.e. non-Dataverse OAI servers, have been fixed. diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index cc4b03bbc20..b14f769f20b 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -169,6 +169,10 @@ OAI-PMH error handling has been improved to display a machine-readable error in A bug introduced with the guestbook-at-request, requests are not deleted when granted, they are now given the state granted. +### Harvesting redirects fixed + +Redirects from search cards back to the original source for datasets harvested from "Generic OAI Archives", i.e. non-Dataverse OAI servers, have been fixed. + [⬆️](#table-of-contents) ## 💾 Persistence From 9d11921f86a23b646972b1ef6775c8cfa63fb612 Mon Sep 17 00:00:00 2001 From: Stephen Kraffmiller Date: Fri, 29 Mar 2024 11:08:52 -0400 Subject: [PATCH 58/61] #10422 fix wording of note --- doc/release-notes/6.2-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index b14f769f20b..5ca03e3231c 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -98,7 +98,7 @@ For more information, see #10382. ### Harvesting Handle Missing Controlled Values -Allows datasets to be harvested with Controlled Vocabulary Values that existed in the originating Dataverse Project but are not in the harvesting Dataverse Project. For more information, view the changes to the endpoint [here](#harvesting-client-endpoint-extended). +Allows datasets to be harvested with Controlled Vocabulary Values that existed in the originating Dataverse installation but are not in the harvesting Dataverse installation. For more information, view the changes to the endpoint [here](#harvesting-client-endpoint-extended). ### Add .QPJ and .QMD Extensions to Shapefile Handling From 934bdf128eed12eb2dc1be56ee1b0b9674e39bca Mon Sep 17 00:00:00 2001 From: landreev Date: Fri, 29 Mar 2024 12:20:37 -0400 Subject: [PATCH 59/61] Update 6.2-release-notes.md --- doc/release-notes/6.2-release-notes.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index 5ca03e3231c..2b95568105a 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -387,14 +387,14 @@ In the following commands we assume that Payara 6 is installed in `/usr/local/pa (or `setenv PAYARA /usr/local/payara6` if you are using a `csh`-like shell) -1\. Usually, when a Solr schema update is released, we recommend deploying the new version of Dataverse, then updating the `schema.xml` on the solr side. With 6.2, we recommend to install the base schema first. Without it Dataverse 6.2 is not going to be able to show any results after the initial deployment. If your instance is using any custom metadata blocks, you will need to further modify the schema, see the laset step of this instruction (step 8). +1\. Usually, when a Solr schema update is released, we recommend deploying the new version of Dataverse, then updating the `schema.xml` on the solr side. With 6.2, we recommend to install the base schema first. Without it Dataverse 6.2 is not going to be able to show any results after the initial deployment. If your instance is using any custom metadata blocks, you will need to further modify the schema, see the last step of this instruction (step 8). - Stop Solr instance (usually `service solr stop`, depending on Solr installation/OS, see the [Installation Guide](https://guides.dataverse.org/en/6.2/installation/prerequisites.html#solr-init-script)) - Replace schema.xml - - `cd /usr/local/solr/solr-9.3.0/server/solr/collection1/conf` - `wget https://raw.githubusercontent.com/IQSS/dataverse/master/conf/solr/9.3.0/schema.xml` + - `cp schema.xml /usr/local/solr/solr-9.3.0/server/solr/collection1/conf` - Start Solr instance (usually `service solr start`, depending on Solr/OS) From 91a9ae95400ae02556ad7dbcef6335b9c2d12e85 Mon Sep 17 00:00:00 2001 From: landreev Date: Fri, 29 Mar 2024 13:44:06 -0400 Subject: [PATCH 60/61] Update 6.2-release-notes.md --- doc/release-notes/6.2-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index 2b95568105a..a80ee8d79e9 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -422,7 +422,7 @@ As noted above, deployment of the war file might take several minutes due a data - `service payara stop` - `service payara start` -7\. Update the following Metadata Blocks: +7\. Update the following Metadata Blocks to reflect the incremental improvements made to the handling of core metadata fields: ``` wget https://github.com/IQSS/dataverse/releases/download/v6.2/geospatial.tsv From 58d81648304949fce1f936e1a631c1147f93a3c0 Mon Sep 17 00:00:00 2001 From: Stephen Kraffmiller Date: Fri, 29 Mar 2024 15:27:17 -0400 Subject: [PATCH 61/61] #10422 wording change --- doc/release-notes/6.2-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.2-release-notes.md b/doc/release-notes/6.2-release-notes.md index a80ee8d79e9..f694703f0a6 100644 --- a/doc/release-notes/6.2-release-notes.md +++ b/doc/release-notes/6.2-release-notes.md @@ -277,7 +277,7 @@ For more information visit the full native API guide on [this link](https://guid ### Endpoint Fixed: Datasets Metadata -The API endpoint `api/datasets/{id}/metadata` has been changed to default to the latest version of the dataset that the user has access. +The API endpoint `api/datasets/{id}/metadata` has been changed to default to the latest version of the dataset to which the user has access. ### Experimental Make Data Count processingState API