Merge branch 'develop' of https://github.com/IQSS/dataverse into mast…

…er_json_fix

.github/ISSUE_TEMPLATE/bug_report.md

@@ -14,7 +14,7 @@ Thank you for contributing to the Dataverse Project through the creation of a bu
 WARNING: If this is a security issue it should be reported privately to [email protected]
 
 More information on bug issues and contributions can be found in the "Contributing to Dataverse" page:
-https://github.com/IQSS/dataverse/blob/develop/CONTRIBUTING.md#bug-reportsissues
+https://guides.dataverse.org/en/latest/contributor/index.html
 
 Please fill out as much of the template as you can.
 Start below this comment section.
@@ -44,7 +44,6 @@ Start below this comment section.
 **Any related open or closed issues to this bug report?**
 
 
-
 **Screenshots:**
 
 No matter the issue, screenshots are always welcome.
@@ -53,3 +52,7 @@ To add a screenshot, please use one of the following formats and/or methods desc
 
 * https://help.github.com/en/articles/file-attachments-on-issues-and-pull-requests
 *
+
+
+**Are you thinking about creating a pull request for this issue?**
+Help is always welcome, is this bug something you or your organization plan to fix?

.github/ISSUE_TEMPLATE/feature_request.md

@@ -1,7 +1,7 @@
 ---
 name: Feature request
 about: Suggest an idea or new feature for the Dataverse software!
-title: 'Feature Request/Idea:'
+title: 'Feature Request:'
 labels: 'Type: Feature'
 assignees: ''
 
@@ -11,7 +11,7 @@ assignees: ''
 Thank you for contributing to the Dataverse Project through the creation of a feature request!
 
 More information on ideas/feature requests and contributions can be found in the "Contributing to Dataverse" page:
-https://github.com/IQSS/dataverse/blob/develop/CONTRIBUTING.md#ideasfeature-requests
+https://guides.dataverse.org/en/latest/contributor/index.html
 
 Please fill out as much of the template as you can.
 Start below this comment section.
@@ -34,3 +34,6 @@ Start below this comment section.
 
 
 **Any open or closed issues related to this feature request?**
+
+**Are you thinking about creating a pull request for this feature?**  
+Help is always welcome, is this feature something you or your organization plan to implement?

.github/ISSUE_TEMPLATE/idea_proposal.md

@@ -0,0 +1,40 @@
+---
+name: Idea proposal
+about: Propose a new idea for discussion to improve the Dataverse software!
+title: 'Suggestion:'
+labels: 'Type: Suggestion'
+assignees: ''
+
+---
+
+<!--
+Thank you for contributing to the Dataverse Project through the creation of a feature request!
+
+More information on ideas/feature requests and contributions can be found in the "Contributing to Dataverse" page:
+https://guides.dataverse.org/en/latest/contributor/index.html
+
+Please fill out as much of the template as you can.
+Start below this comment section.
+-->
+
+**Overview of the Suggestion**
+
+
+**What kind of user is the suggestion intended for?**
+(Example users roles: API User, Curator, Depositor, Guest, Superuser, Sysadmin)
+
+
+**What inspired this idea?**
+
+
+**What existing behavior do you want changed?**
+
+
+**Any brand new behavior do you want to add to Dataverse?**
+
+
+**Any open or closed issues related to this suggestion?**
+
+
+**Are you thinking about creating a pull request for this issue?**
+Help is always welcome, is this idea something you or your organization plan to implement?

.github/workflows/check_property_files.yml

@@ -0,0 +1,32 @@
+name: "Properties Check"
+on:
+    pull_request:
+        paths:
+            - "src/**/*.properties"
+            - "scripts/api/data/metadatablocks/*"
+jobs:
+    duplicate_keys:
+        name: Duplicate Keys
+        runs-on: ubuntu-latest
+        steps:
+            - uses: actions/checkout@v4
+            - name: Run duplicates detection script
+              shell: bash
+              run: tests/check_duplicate_properties.sh
+
+    metadata_blocks_properties:
+        name: Metadata Blocks Properties
+        runs-on: ubuntu-latest
+        steps:
+            - uses: actions/checkout@v4
+            - name: Setup GraalVM + Native Image
+              uses: graalvm/setup-graalvm@v1
+              with:
+                github-token: ${{ secrets.GITHUB_TOKEN }}
+                java-version: '21'
+                distribution: 'graalvm-community'
+            - name: Setup JBang
+              uses: jbangdev/setup-jbang@main
+            - name: Run metadata block properties verification script
+              shell: bash
+              run: tests/verify_mdb_properties.sh

.github/workflows/shellspec.yml

@@ -24,28 +24,11 @@ jobs:
               run: |
                   cd tests/shell
                   shellspec
-    shellspec-centos7:
-        name: "CentOS 7"
+    shellspec-rocky9:
+        name: "RockyLinux 9"
         runs-on: ubuntu-latest
         container:
-            image: centos:7
-        steps:
-            - uses: actions/checkout@v2
-            - name: Install shellspec
-              run: |
-                  curl -fsSL https://github.com/shellspec/shellspec/releases/download/${{ env.SHELLSPEC_VERSION }}/shellspec-dist.tar.gz | tar -xz -C /usr/share
-                  ln -s /usr/share/shellspec/shellspec /usr/bin/shellspec
-            - name: Install dependencies
-              run: yum install -y ed
-            - name: Run shellspec
-              run: |
-                  cd tests/shell
-                  shellspec
-    shellspec-rocky8:
-        name: "RockyLinux 8"
-        runs-on: ubuntu-latest
-        container:
-            image: rockylinux/rockylinux:8
+            image: rockylinux/rockylinux:9
         steps:
             - uses: actions/checkout@v2
             - name: Install shellspec

doc/release-notes/10169-JSON-schema-validation.md

@@ -0,0 +1,3 @@
+### Improved JSON Schema validation for datasets
+
+Enhanced JSON schema validation with checks for required and allowed child objects, type checking for field types including `primitive`, `compound` and `controlledVocabulary`. More user-friendly error messages to help pinpoint the issues in the dataset JSON. See [Retrieve a Dataset JSON Schema for a Collection](https://guides.dataverse.org/en/6.3/api/native-api.html#retrieve-a-dataset-json-schema-for-a-collection) in the API Guide and PR #10543.

doc/release-notes/10341-croissant.md

@@ -0,0 +1,9 @@
+A new metadata export format called Croissant is now available as an external metadata exporter. It is oriented toward making datasets consumable by machine learning.
+
+When enabled, Croissant replaces the Schema.org JSON-LD format in the `<head>` of dataset landing pages. For details, see the [Schema.org JSON-LD/Croissant Metadata](https://dataverse-guide--10533.org.readthedocs.build/en/10533/admin/discoverability.html#schema-org-head) under the discoverability section of the Admin Guide.
+
+For more about the Croissant exporter, see https://github.com/gdcc/exporter-croissant
+
+For installation instructions, see [Enabling External Exporters](https://dataverse-guide--10533.org.readthedocs.build/en/10533/installation/advanced.html#enabling-external-exporters) in the Installation Guide.
+
+See also Issue #10341 and PR #10533.

doc/release-notes/10508-base-image-fixes.md

@@ -0,0 +1,12 @@
+# Security and Compatibility Fixes to the Container Base Image
+
+- Switch "wait-for" to "wait4x", aligned with the Configbaker Image
+- Update "jattach" to v2.2
+- Install AMD64 / ARM64 versions of tools as necessary
+- Run base image as unprivileged user by default instead of `root` - this was an oversight from OpenShift changes
+- Linux User, Payara Admin and Domain Master passwords:
+  - Print hints about default, public knowledge passwords in place for
+  - Enable replacing these passwords at container boot time
+- Enable building with updates Temurin JRE image based on Ubuntu 24.04 LTS
+- Fix entrypoint script troubles with pre- and postboot script files
+- Unify location of files at CONFIG_DIR=/opt/payara/config, avoid writing to other places

doc/release-notes/10606-dataverse-in-windows-wsl.md

@@ -0,0 +1 @@
+New instructions have been added for developers on Windows trying to run a Dataverse development environment using Windows Subsystem for Linux (WSL). See https://dataverse-guide--10608.org.readthedocs.build/en/10608/developers/windows.html #10606 and #10608.

doc/release-notes/10633-add-dataverse-api-extension.md

@@ -0,0 +1 @@
+The addDataverse (/api/dataverses/{identifier}) API endpoint has been extended to allow adding metadata blocks, input levels and facet ids at creation time, as the Dataverse page in create mode does in JSF.

doc/release-notes/10726-dataverse-facets-api-extension.md

@@ -0,0 +1,3 @@
+New optional query parameter "returnDetails" added to "dataverses/{identifier}/facets/" endpoint to include detailed information of each DataverseFacet.
+
+New endpoint "datasetfields/facetables" that lists all facetable dataset fields defined in the installation.

doc/release-notes/10733-add-publication-status-to-search-api-results.md

@@ -0,0 +1,14 @@
+Search API (/api/search) response will now include publicationStatuses in the Json response as long as the list is not empty
+
+Example:
+```javascript
+"items": [
+    {
+        "name": "Darwin's Finches",
+        ...
+        "publicationStatuses": [
+            "Unpublished",
+            "Draft"
+        ],
+(etc, etc)
+```

doc/release-notes/10741-list-metadatablocks-display-on-create-fix.md

@@ -0,0 +1 @@
+Fixed dataverses/{identifier}/metadatablocks endpoint to not return fields marked as displayOnCreate=true if there is an input level with include=false, when query parameters returnDatasetFieldTypes=true and onlyDisplayedOnCreate=true are set.

doc/release-notes/10749-dataverse-user-permissions-api-extension.md

@@ -0,0 +1 @@
+New API endpoint "dataverses/{identifier}/userPermissions" for obtaining the user permissions on a dataverse.

doc/release-notes/10758-rust-client.md

@@ -0,0 +1,3 @@
+### Rust API client library
+
+An API client library for the Rust programming language is now available at https://github.com/gdcc/rust-dataverse and has been added to the [list of client libraries](https://dataverse-guide--10758.org.readthedocs.build/en/10758/api/client-libraries.html) in the API Guide. See also #10758.

doc/release-notes/7068-reserve-file-pids.md

@@ -0,0 +1,9 @@
+## Release Highlights
+
+### Pre-Publish File DOI Reservation with DataCite
+
+Dataverse installations using DataCite (or other persistent identifier (PID) Providers that support reserving PIDs) will be able to reserve PIDs for files when they are uploaded (rather than at publication time). Note that reserving file DOIs can slow uploads with large numbers of files so administrators may need to adjust timeouts (specifically any Apache "``ProxyPass / ajp://localhost:8009/ timeout=``" setting in the recommended Dataverse configuration).
+
+## Major Use Cases
+
+- Users will have DOIs/PIDs reserved for their files as part of file upload instead of at publication time. (Issue #7068, PR #7334)

doc/release-notes/8796-fix-license-display-indexing.md

@@ -0,0 +1 @@
+When datasets have neither a license nor custom terms of use the display will indicate this. Also, these datasets will no longer be indexed as having custom terms. 

doc/release-notes/8945-ignore-shapefiles-under-hidden-directories-in-zip.md

@@ -0,0 +1,5 @@
+### Shapefile Handling will now ignore files under a hidden directory within the zip file
+
+Directories that are hidden will be ignored when determining if a zip file contains Shapefile files.
+
+For more information, see #8945.

doc/release-notes/9081-CC0-waiver-turned-into-custom-license.md

@@ -0,0 +1,6 @@
+In an earlier Dataverse release, Datasets with only 'CC0 Waiver' in termsofuse field were converted to 'Custom License' instead of CC0 1.0 licenses during an automated process. A new process was added to correct this. Only Datasets with no terms other than the one create by the previous process will be modified.
+- The existing 'Terms of Use' must be equal to 'This dataset is made available under a Creative Commons CC0 license with the following additional/modified terms and conditions: CC0 Waiver'
+- The following terms fields must be empty: Confidentiality Declaration, Special Permissions, Restrictions, Citation Requirements, Depositor Requirements, Conditions, and Disclaimer.
+- The License ID must not be assigned.
+
+This process will set the License ID to that of the CC0 1.0 license and remove the contents of termsofuse field.

doc/release-notes/api-blocking-filter-json.md

@@ -0,0 +1,3 @@
+* When any `ApiBlockingFilter` policy applies to a request, the JSON in the body of the error response is now valid JSON.
+  In case an API client did any special processing to allow it to parse the body, that is no longer necessary.
+  The status code of such responses has not changed.

doc/release-notes/make-data-count-.md

@@ -0,0 +1,3 @@
+### Counter Processor 1.05 Support
+
+This release includes support for counter-processor-1.05 for processing Make Data Count metrics. If you are running Make Data Counts support, you should reinstall/reconfigure counter-processor as described in the latest Guides. (For existing installations, note that counter-processor-1.05 requires a Python3, so you will need to follow the full counter-processor install. Also note that if you configure the new version the same way, it will reprocess the days in the current month when it is first run. This is normal and will not affect the metrics in Dataverse.)

doc/sphinx-guides/source/_static/api/dataverse-complete-optional-params.json

@@ -0,0 +1,65 @@
+{
+  "name": "Scientific Research",
+  "alias": "science",
+  "dataverseContacts": [
+    {
+      "contactEmail": "[email protected]"
+    },
+    {
+      "contactEmail": "[email protected]"
+    }
+  ],
+  "affiliation": "Scientific Research University",
+  "description": "We do all the science.",
+  "dataverseType": "LABORATORY",
+  "metadataBlocks": {
+    "metadataBlockNames": [
+      "citation", "geospatial"
+    ],
+    "inputLevels": [
+      {
+        "datasetFieldTypeName": "geographicCoverage",
+        "include": true,
+        "required": true
+      },
+      {
+        "datasetFieldTypeName": "country",
+        "include": true,
+        "required": true
+      },
+      {
+        "datasetFieldTypeName": "geographicUnit",
+        "include": false,
+        "required": false
+      },
+      {
+        "datasetFieldTypeName": "geographicBoundingBox",
+        "include": false,
+        "required": false
+      },
+      {
+        "datasetFieldTypeName": "westLongitude",
+        "include": false,
+        "required": false
+      },
+      {
+        "datasetFieldTypeName": "eastLongitude",
+        "include": false,
+        "required": false
+      },
+      {
+        "datasetFieldTypeName": "northLatitude",
+        "include": false,
+        "required": false
+      },
+      {
+        "datasetFieldTypeName": "southLatitude",
+        "include": false,
+        "required": false
+      }
+    ],
+    "facetIds": [
+      "authorName", "authorAffiliation"
+    ]
+  }
+}

doc/sphinx-guides/source/_static/util/counter_daily.sh

@@ -1,6 +1,6 @@
 #! /bin/bash
 
-COUNTER_PROCESSOR_DIRECTORY="/usr/local/counter-processor-0.1.04"
+COUNTER_PROCESSOR_DIRECTORY="/usr/local/counter-processor-1.05"
 MDC_LOG_DIRECTORY="/usr/local/payara6/glassfish/domains/domain1/logs/mdc"
 
 # counter_daily.sh

doc/sphinx-guides/source/_templates/navbar.html

@@ -25,7 +25,6 @@
                     <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">About <span class="caret"></span></a>
                     <ul class="dropdown-menu">
                         <li><a target="_blank" href="https://dataverse.org/about">About the Project</a></li>
-                        <li><a target="_blank" href="https://dataverse.org/add-data">Add Data</a></li>
                         <li><a target="_blank" href="https://dataverse.org/blog">Blog</a></li>
                         <li><a target="_blank" href="https://dataverse.org/presentations">Presentations</a></li>
                         <li><a target="_blank" href="https://dataverse.org/publications">Publications</a></li>

doc/sphinx-guides/source/admin/discoverability.rst

@@ -30,14 +30,21 @@ The HTML source of a dataset landing page includes "DC" (Dublin Core) ``<meta>``
         <meta name="DC.type" content="Dataset"
         <meta name="DC.title" content="..."
 
-Schema.org JSON-LD Metadata
-+++++++++++++++++++++++++++
+.. _schema.org-head:
 
-The HTML source of a dataset landing page includes Schema.org JSON-LD metadata like this::
+Schema.org JSON-LD/Croissant Metadata
++++++++++++++++++++++++++++++++++++++
+
+The ``<head>`` of the HTML source of a dataset landing page includes Schema.org JSON-LD metadata like this::
 
 
         <script type="application/ld+json">{"@context":"http://schema.org","@type":"Dataset","@id":"https://doi.org/...
 
+If you enable the Croissant metadata export format (see :ref:`external-exporters`) the ``<head>`` will show Croissant metadata instead. It looks similar, but you should see ``"cr": "http://mlcommons.org/croissant/"`` in the output.
+
+For backward compatibility, if you enable Croissant, the older Schema.org JSON-LD format (``schema.org`` in the API) will still be available from both the web interface (see :ref:`metadata-export-formats`) and the API (see :ref:`export-dataset-metadata-api`).
+
+The Dataverse team has been working with Google on both formats. Google has `indicated <https://github.com/mlcommons/croissant/issues/530#issuecomment-1964227662>`_ that for `Google Dataset Search <https://datasetsearch.research.google.com>`_ (the main reason we started adding this extra metadata in the ``<head>`` of dataset pages), Croissant is the successor to the older format.
 
 .. _discovery-sign-posting:
 

doc/sphinx-guides/source/admin/make-data-count.rst

@@ -16,7 +16,7 @@ Architecture
 
 Dataverse installations who would like support for Make Data Count must install `Counter Processor`_, a Python project created by California Digital Library (CDL) which is part of the Make Data Count project and which runs the software in production as part of their `DASH`_ data sharing platform.
 
-.. _Counter Processor: https://github.com/CDLUC3/counter-processor
+.. _Counter Processor: https://github.com/gdcc/counter-processor
 .. _DASH: https://cdluc3.github.io/dash/
 
 The diagram below shows how Counter Processor interacts with your Dataverse installation and the DataCite hub, once configured. Dataverse installations using Handles rather than DOIs should note the limitations in the next section of this page.
@@ -84,9 +84,9 @@ Configure Counter Processor
 
 * Change to the directory where you installed Counter Processor.
 
-  * ``cd /usr/local/counter-processor-0.1.04``
+  * ``cd /usr/local/counter-processor-1.05``
 
-* Download :download:`counter-processor-config.yaml <../_static/admin/counter-processor-config.yaml>` to ``/usr/local/counter-processor-0.1.04``.
+* Download :download:`counter-processor-config.yaml <../_static/admin/counter-processor-config.yaml>` to ``/usr/local/counter-processor-1.05``.
 
 * Edit the config file and pay particular attention to the FIXME lines.
 
@@ -99,7 +99,7 @@ Soon we will be setting up a cron job to run nightly but we start with a single
 
 * Change to the directory where you installed Counter Processor.
 
-  * ``cd /usr/local/counter-processor-0.1.04``
+  * ``cd /usr/local/counter-processor-1.05``
 
 * If you are running Counter Processor for the first time in the middle of a month, you will need create blank log files for the previous days. e.g.:
 

doc/sphinx-guides/source/api/client-libraries.rst

@@ -78,3 +78,10 @@ Ruby
 https://github.com/libis/dataverse_api is a Ruby gem for Dataverse APIs. It is registered as a library on Rubygems (https://rubygems.org/search?query=dataverse).
 
 The gem is created and maintained by the LIBIS team (https://www.libis.be) at the University of Leuven (https://www.kuleuven.be).
+
+Rust
+----
+
+https://github.com/gdcc/rust-dataverse
+
+The Rust Dataverse client is a comprehensive crate designed for seamless interaction with the Dataverse API. It facilitates essential operations such as collection, dataset, and file management. Additionally, the crate includes a user-friendly command-line interface (CLI) that brings the full functionality of the library to the command line. This project is actively maintained by `Jan Range <https://github.com/jr-1991>`_.