diff --git a/Functional requirements.md b/Functional requirements.md index 94deabc..fc40650 100644 --- a/Functional requirements.md +++ b/Functional requirements.md @@ -15,7 +15,7 @@ and an `OutputStream` to which the output is written as its arguments and return ## New design The new design of `easy-deposit-agreement-creator` consists of (1) an API which can be called from within other -modules that are dependents of this module and (2) a command line tool. In case the latter is used, +modules which depend on this module and (2) a command line tool. In case the latter is used, we assume that the dataset as well as the depositor data are already present in EASY. Again notice that this command line tool is not intended to ingest the newly generated deposit agreement into EASY! @@ -24,7 +24,7 @@ The input and output of both parts of `easy-deposit-agreement-creator` are as fo * *input (via API call):* either one of * dataset identifier, `OutputStream` - *used in modification tools* * EMD object, depositor object, `OutputStream` - *used in the business-layer* - * EMD object, depositor identifier, list of file metadata objects, `OutputStream` - *used in Easy-Stage-Dataset* + * EMD object, depositor identifier, an obsolete list of file metadata objects, `OutputStream` - *used in Easy-Stage-Dataset* * dataset object, `OutputStream` * *output (via command line):* pdf document with the deposit agreement * *output (via API call):* `Unit` @@ -40,29 +40,23 @@ The input and output of both parts of `easy-deposit-agreement-creator` are as fo * `easy-deposit-agreement-creator` wil *not* send emails to depositors whose datasets are modified or newly ingested. * No data is written to the databases; this module only reads data! -### Additions relative to the former version - * In the list of files in the dataset the access category for each file needs to be included. - * An explaination of the distinct access categories from the previous item. - ## Resources The deposit agreement is generated from a series of template files with placeholders. Using the resources listed below this module can resolve these placeholders and transform the whole text into a pdf. ### Template files - * `Appendix.html` - an appendix with more information about the CC0 access category + * `styles.css` - styling for the various elements + * `Agreement.html` - the main template with the content for the footer and parsing the other templates * `Body.html` - the main content of the deposit agreement text + * `Appendix1.html` - an appendix with the chosen access rights, license and an optional embargo statement + * `Appendix2.html` - an appendix with the dans license that may be applicable or not as explained in the body.html * `dans_logo.png` - the Dans logo to be displayed in the header of each page - * `Embargo.html` - an optional text in case the dataset is under embargo - * `FileTable.html` - an overview of all the files in the dataset, showing the file path, checksum and access category - * `Agreement.html` - the main file in which all the other html files are merged together - * `agreement_version.txt` - the version of the agreement to be displayed in the footer of each page of the document + * `DriveByData.png` - background in the footer * `MetadataTerms.properties` - a mapping between terms from the metadata and the text to be displayed in the agreement - * `Table.html` - an overview of the metadata of the dataset ### Data resources - * *Fedora* - metadata of the dataset is stored in Fedora. The EMD datastream dissemination contains the metadata of the dataset itself, the AMD datastream dissemination contains the depositor identifier, the EASY_FILE datastream and EASY_FILE_METADATA datastream dissemination contain the data of the files in the dataset. + * *Fedora* - metadata of the dataset is stored in Fedora. The EMD datastream dissemination contains the metadata of the dataset itself, the AMD datastream dissemination contains the depositor identifier. * *LDAP* - the depositor data required for the agreement is stored in LDAP. - * *RiSearch* - this is part of Fedora and provides the relation between the dataset and the files. ### Required data in the template Besides the dataset's metadata and the list of files contained in the dataset, several other values @@ -71,14 +65,13 @@ are required in the creation of the deposit agreement. | Data | Used in | Stored in | |------|---------|-----------| | Dataset - identifier | all occasions where a query for (a part of) the dataset in Fedora is required | application parameter | -| Dataset - DANS managed DOI | template `Body.html` | `emd:identifier // dc:identifier` | -| Dataset - encoded DANS managed DOI | template `Body.html`, see the link on the managed DOI above | `let id = emd:identifier // dc:identifier in (id@eas:identification-system ++ "/" ++ id.value)` | -| Dataset - date submitted | template `Body.html` | `emd:date // eas:dateSubmitted` | -| Dataset - preferred title | template `Body.html` | `emd:title // dc:title` | -| Dataset - access category | template `Agreement.html` | `emd:rights // dct:accessRights` or `dc:rights` (*these are always the same, only in different schemas. Therefore we can always use the value from EMD to get the least amount of Fedora queries*)| -| Dataset - is under embargo | code `LicenseComposer.java:193` | to be calculated based on the current date and `Dataset - date available` below | -| Dataset - date available | template `Embargo.html` | `emd:date // eas:available` | -| Current time | template `Tail.html`, this is the timestamp of creating the deposit agreement: `new org.joda.time.DateTime().toString("YYYY-MM-dd HH:mm:ss"))` | calculated at runtime | +| Dataset - DANS managed DOI | template `Header.tml` | `emd:identifier // dc:identifier` | +| Dataset - encoded DANS managed DOI | template `Header.html`, see the link on the managed DOI above | `let id = emd:identifier // dc:identifier in (id@eas:identification-system ++ "/" ++ id.value)` | +| Dataset - date submitted | template `Header.tml` | `emd:date // eas:dateSubmitted` | +| Dataset - preferred title | template `Header.html` | `emd:title // dc:title` | +| Dataset - open access | template `Apppendix1.html` | `emd:rights // dct:accessRights` or `dc:rights`
(*these are always the same, only in different schemas. Therefore we can always use the value from EMD to get the least amount of Fedora queries*)| +| Dataset - is under embargo | template `Apppendix1.html` | to be calculated based on the current date and `Dataset - date available` below | +| Dataset - date available | template `Apppendix1.html` | `emd:date // eas:available` | | EasyUser - displayName | template `Body.html` | LDAP user database - `(givenName <> initials)? + dansPrefixes? + sn?` | | EasyUser - organization | template `Body.html` | LDAP user database - `o` | | EasyUser - address | template `Body.html` | LDAP user database - `postalAddress` | @@ -88,32 +81,6 @@ are required in the creation of the deposit agreement. | EasyUser - telephone | template `Body.html` | LDAP user database - `telephoneNumber` | | EasyUser - email | template `Body.html` | LDAP user database - `mail` | -### Displaying the dataset metadata -* For each term in the metadata the *qualified name* is calculated ([namespace].[name]) and mapped to the corresponding value in `MetadataTerms.properties`. -* If the term equals **AUDIENCE**, all associated *discipline identifiers* are queried from Fedora and displayed as a comma-separated `String`. -* If the term equals **ACCESSRIGHTS**, it is mapped to the corresponding string representation (see below). -* For all other terms the values are displayed as a comma-separated `String`. -* Every term corresponds to one row in the table. - -### Displaying the files in the dataset -* All files contained in the dataset are retrieved from Fedora using `RiSearch`. -* For each file the SHA1-hash is queried. -* Each file (path and hash) corresponds to one row in the table. -* In case the dataset does not contain any files, the text "*No uploaded files*" is added instead of the table. -* In case the SHA1-hash of a file is not calculated, the alternative text "*------------- not-calculated -------------*" is used - -### Mapping of access categories -| Access Category | Agreement snippet | String representation | -|-----------------|-----------------|-----------------------| -| ANONYMOUS_ACCESS | OpenAccess.html | `"Anonymous"` | -| OPEN_ACCESS | OpenAccess.html | `"Open Access"` | -| OPEN_ACCESS_FOR_REGISTERED_USERS | OpenAccessForRegisteredUsers.html | `"Open access for registered users"` | -| GROUP_ACCESS | RestrictGroup.html | `"Restricted -'archaeology' group"` | -| REQUEST_PERMISSION | RestrictRequest.html | `"Restricted -request permission"` | -| ACCESS_ELSEWHERE | OtherAccess.html | `"Elsewhere"` | -| NO_ACCESS | OtherAccess.html | `"Other"` | -| FREELY_AVAILABLE | OpenAccess.html | `"Open Access"` | - ## Page layout * The document has an A4 page size and the following margins (top-right-bottom-left): 2.5cm 1.5cm 2cm 1.5cm * Every page has a header with the DANS logo @@ -134,12 +101,11 @@ resolved. This only happens when the property `runtime.references.strict = true` Velocity properties file. Besides that Velocity requires the path to the resources to be set using the property `file.resource.loader.path`. As an extra parameter we added `template.file.name`, holding the name of the file to be resolved by Velocity. This file is supposed to be present inside -the `file.resource.loader.path` folder. All these parameters can are set in the -`velocity-engine.properties` file in `src/main/assembly/dist/res/`. +the `file.resource.loader.path` folder. All these parameters are [hard coded](https://github.com/DANS-KNAW/easy-deposit-agreement-creator/blob/230a1e1ffaf24213f71277c1de1dbb9cd08daf96/src/main/scala/nl/knaw/dans/easy/agreement/internal/package.scala#L43-L51). ### WeasyPrint The transformation from html to pdf is done using the WeasyPrint command line tool. This tool is -installed on the servers (*deasy*, *teasy* and *easy01*). For running it locally (during -development) we recommend using the `localrun.sh` script. Indicate this in the +installed on the servers (*deasy*, *teasy* and *easy11*). For running it locally (during +development) we recommend using the `src/main/assembly/dist/res/pdfgen.sh` script. Indicate this in the `application.properties` located in `src/main/assembly/dist/cfg/` and fill in the `...` placeholders in the script. diff --git a/debug-init-env.sh b/debug-init-env.sh index 8e90f4c..ee6d85e 100755 --- a/debug-init-env.sh +++ b/debug-init-env.sh @@ -20,3 +20,4 @@ DATADIR=data touch $DATADIR/easy-deposit-agreement-creator.log cp src/test/resources/debug-config/pdfgen.sh $HOMEDIR/res/ +cp -r target/easy-licenses/licenses $HOMEDIR/res/licenses \ No newline at end of file diff --git a/docs/01_manual.md b/docs/01_manual.md new file mode 100755 index 0000000..e69de29 diff --git a/docs/index.md b/docs/index.md index 753ded8..b96c99b 100755 --- a/docs/index.md +++ b/docs/index.md @@ -70,6 +70,10 @@ yum install redhat-rpm-config python-devel python-pip python-lxml cairo pango gd After this, `weasyprint --help` is supposed to show the appropriate help page. +Python however may complain about `unknown locale`, add to your profile: + + export LC_ALL=en_US.UTF-8 + export LANG=en_US.UTF-8 BUILDING FROM SOURCE -------------------- diff --git a/pom.xml b/pom.xml index 78ebec0..5bdbd1a 100644 --- a/pom.xml +++ b/pom.xml @@ -37,6 +37,7 @@ nl.knaw.dans.easy.agreement.Command 3.8.0 + 1.0.3 @@ -175,6 +176,33 @@ + + + + maven-dependency-plugin + + + resources + generate-resources + + unpack + + + + + nl.knaw.dans.easy + easy-licenses + ${easy-licenses.version} + ${project.build.directory}/easy-licenses + + + + + + + + + rpm @@ -219,6 +247,24 @@ + + /var/opt/${dans-provider-name}/resource/${project.artifactId}/template/licenses + ${rpm-replace-configuration} + + + target/easy-licenses/licenses + + + + + /opt/${dans-provider-name}/${project.artifactId}/res/template/licenses + ${rpm-replace-configuration} + + + target/easy-licenses/licenses + + + diff --git a/src/main/assembly/dist/cfg/application.properties b/src/main/assembly/dist/cfg/application.properties index 25718bf..eb99e65 100644 --- a/src/main/assembly/dist/cfg/application.properties +++ b/src/main/assembly/dist/cfg/application.properties @@ -1,12 +1,8 @@ fcrepo.url=http://localhost:8080/fedora fcrepo.user=fedoraAdmin fcrepo.password=changeme -fsrdb.db-connection-url=jdbc:postgresql://localhost:5432/easy_db -fsrdb.db-connection-username=easy_webui -fsrdb.db-connection-password=changeme auth.ldap.url=ldap://localhost auth.ldap.user=cn=ldapadmin,dc=dans,dc=knaw,dc=nl auth.ldap.password=changeme agreement.resources=/var/opt/dans.knaw.nl/resource/easy-deposit-agreement-creator -agreement.fileLimit=500 daemon.http.port=20130 diff --git a/src/main/assembly/dist/res/DrivenByData.png b/src/main/assembly/dist/res/DrivenByData.png new file mode 100644 index 0000000..a2f5814 Binary files /dev/null and b/src/main/assembly/dist/res/DrivenByData.png differ diff --git a/src/main/assembly/dist/res/agreement_version.txt b/src/main/assembly/dist/res/agreement_version.txt deleted file mode 100644 index 71fc71e..0000000 --- a/src/main/assembly/dist/res/agreement_version.txt +++ /dev/null @@ -1 +0,0 @@ -Version 5.3 EN; 27-06-2016 diff --git a/src/main/assembly/dist/res/dans_logo.png b/src/main/assembly/dist/res/dans_logo.png index 7402678..7fb097d 100644 Binary files a/src/main/assembly/dist/res/dans_logo.png and b/src/main/assembly/dist/res/dans_logo.png differ diff --git a/src/main/assembly/dist/res/template/Agreement.html b/src/main/assembly/dist/res/template/Agreement.html index 0ef492e..aa95ff5 100644 --- a/src/main/assembly/dist/res/template/Agreement.html +++ b/src/main/assembly/dist/res/template/Agreement.html @@ -2,57 +2,7 @@ @@ -64,48 +14,39 @@ #end #parse("Body.html") - -#if ($OpenAccess) -

[Open Access: unlimited access without registration of user registration]

-

The Depositor agrees to the dataset being made available in accordance with the conditions of the Creative Commons Zero Waiver, the CC0 1.0 Universal Public Domain Dedication (Appendix 1). In doing so, the Depositor renounces all possible rights relating to the dataset.

-#elseif ($OpenAccessForRegisteredUsers) -

[Open Access for Registered Users: unlimited access for registered users]

-

The Repository is permitted to make the dataset available to all persons, legal entities and organisations registered with the Repository.

-#elseif ($OtherAccess) -

[Other Access: the data are not available via EASY]

-

The dataset will be made available by means of another method to be agreed with the Repository.

-#elseif ($RestrictGroup) -

[Restricted Access: access restricted to registered persons or group members, N.B. only for archeology]

-

The Depositor may grant access permission in advance for persons, legal entities and organisations that belong to one of the user groups specified by DANS and/or the Depositor.

-#elseif ($RestrictRequest) -

[Restricted Access: access with the permission of the Repository]

-

The Repository is permitted to make the dataset available to persons, legal entities and organisations registered with the Depositor only after receiving express permission from the Depositor.

-#else - -

NO VALID VALUE FOR THE ACCESS CATEGORY FOUND!!!

-#end - -#if ($UnderEmbargo) -

You have additionally chosen:

-

[Temporary restriction: Embargo]; only possible if Open Access, Open Access for Registered Users or Restricted Access has been chosen

-

The dataset will be temporarily unavailable until $DateAvailable, commencing on the date of publication. The embargo period cannot be longer than two years and cannot be extended. When this period elapses, one of the special provisions set out above shall automatically apply. An extension of this period is only possible in consultation with the Depositor.

+#if (! $IsSample) +

This agreement has been accepted by both parties on $DateSubmitted upon completion of the deposit process via easy.dans.knaw.nl.

#end -

The Depositor hereby agrees to the above provisions and the general code(s) of conduct referred to in this document.

- -#parse("Table.html") - -#if ($HasFiles) -
- #parse("FileTable.html") -#else -

No uploaded files.

+#parse("Appendix1.html") +#parse("Appendix2.html") +#if ($OpenAccess) +

Appendix 3 Legal text of chosen public-domain statement or Open Access Licence

+ 
+        #parse($Appendix3)
+
#end -
-#parse("Appendix.html") +
+

+ Deposit agreement +
+ Version 26-09-2019 +

+

+ DANS promotes sustainable access to digital research data. + See www.dans.knaw.nl for more information. +

+

+ Data Archiving and Networked Services (DANS) +
+ Anna van Saksenlaan 51 | 2593 HW Den Haag +
+ 070 349 44 50 | info@dans.knaw.nl | www.dans.knaw.nl +
+ CoC 54667089 | DANS is an institute of KNAW and NWO +

+
diff --git a/src/main/assembly/dist/res/template/Appendix.html b/src/main/assembly/dist/res/template/Appendix.html deleted file mode 100644 index f437793..0000000 --- a/src/main/assembly/dist/res/template/Appendix.html +++ /dev/null @@ -1,46 +0,0 @@ -

Appendix 1 text CC Zero Waiver

-Source: http://creativecommons.org/publicdomain/zero/1.0/legalcode - -

CC0 1.0 Universal (CC0 1.0) Public Domain Dedication

- -

N.B. The articles mentioned in this appendix are those of the Creative Commons Zero Waiver licence, the CC0 1.0 Universal Public Domain Dedication. This licence is only valid if the access category “Open Access: unlimited access without registration of user registration” has been chosen for disseminating the datasets, or parts of it.

- -

Statement of Purpose

-

The laws of most jurisdictions throughout the world automatically confer exclusive Copyright and Related Rights (defined below) upon the creator and subsequent owner(s) (each and all, an “owner”) of an original work of authorship and/or a database (each, a “Work”).

- -

Certain owners wish to permanently relinquish those rights to a Work for the purpose of contributing to a commons of creative, cultural and scientific works (“Commons”) that the public can reliably and without fear of later claims of infringement build upon, modify, incorporate in other works, reuse and redistribute as freely as possible in any form whatsoever and for any purposes, including without limitation commercial purposes. These owners may contribute to the Commons to promote the ideal of a free culture and the further production of creative, cultural and scientific works, or to gain reputation or greater distribution for their Work in part through the use and efforts of others.

- -

For these and/or other purposes and motivations, and without any expectation of additional consideration or compensation, the person associating CC0 with a Work (the “Affirmer”), to the extent that he or she is an owner of Copyright and Related Rights in the Work, voluntarily elects to apply CC0 to the Work and publicly distribute the Work under its terms, with knowledge of his or her Copyright and Related Rights in the Work and the meaning and intended legal effect of CC0 on those rights.

- -

- 1. Copyright and Related Rights. - A Work made available under CC0 may be protected by copyright and related or neighboring rights (“Copyright and Related Rights”). Copyright and Related Rights include, but are not limited to, the following: -

- -
    -
  1. the right to reproduce, adapt, distribute, perform, display, communicate, and translate a Work;
    2. -
  2. moral rights retained by the original author(s) and/or performer(s);
    3. -
  3. publicity and privacy rights pertaining to a person's image or likeness depicted in a Work;
    4. -
  4. rights protecting against unfair competition in regards to a Work, subject to the limitations in paragraph 4(a), below;
    5. -
  5. rights protecting the extraction, dissemination, use and reuse of data in a Work;
    6. -
  6. database rights (such as those arising under Directive 96/9/EC of the European Parliament and of the Council of 11 March 1996 on the legal protection of databases, and under any national implementation thereof, including any amended or successor version of such directive); and
    7. -
  7. other similar, equivalent or corresponding rights throughout the world based on applicable law or treaty, and any national implementations thereof.
    8. -
- -

- 2. Waiver. - To the greatest extent permitted by, but not in contravention of, applicable law, Affirmer hereby overtly, fully, permanently, irrevocably and unconditionally waives, abandons, and surrenders all of Affirmer's Copyright and Related Rights and associated claims and causes of action, whether now known or unknown (including existing as well as future claims and causes of action), in the Work (i) in all territories worldwide, (ii) for the maximum duration provided by applicable law or treaty (including future time extensions), (iii) in any current or future medium and for any number of copies, and (iv) for any purpose whatsoever, including without limitation commercial, advertising or promotional purposes (the “Waiver”). Affirmer makes the Waiver for the benefit of each member of the public at large and to the detriment of Affirmer's heirs and successors, fully intending that such Waiver shall not be subject to revocation, rescission, cancellation, termination, or any other legal or equitable action to disrupt the quiet enjoyment of the Work by the public as contemplated by Affirmer's express Statement of Purpose. -

- -

- 3. Public License Fallback. - Should any part of the Waiver for any reason be judged legally invalid or ineffective under applicable law, then the Waiver shall be preserved to the maximum extent permitted taking into account Affirmer's express Statement of Purpose. In addition, to the extent the Waiver is so judged Affirmer hereby grants to each affected person a royalty-free, non transferable, non sublicensable, non exclusive, irrevocable and unconditional license to exercise Affirmer's Copyright and Related Rights in the Work (i) in all territories worldwide, (ii) for the maximum duration provided by applicable law or treaty (including future time extensions), (iii) in any current or future medium and for any number of copies, and (iv) for any purpose whatsoever, including without limitation commercial, advertising or promotional purposes (the “License”). The License shall be deemed effective as of the date CC0 was applied by Affirmer to the Work. Should any part of the License for any reason be judged legally invalid or ineffective under applicable law, such partial invalidity or ineffectiveness shall not invalidate the remainder of the License, and in such case Affirmer hereby affirms that he or she will not (i) exercise any of his or her remaining Copyright and Related Rights in the Work or (ii) assert any associated claims and causes of action with respect to the Work, in either case contrary to Affirmer's express Statement of Purpose. -

- -

4. Limitations and Disclaimers.

-
    -
  1. No trademark or patent rights held by Affirmer are waived, abandoned, surrendered, licensed or otherwise affected by this document.
    2. -
  2. Affirmer offers the Work as-is and makes no representations or warranties of any kind concerning the Work, express, implied, statutory or otherwise, including without limitation warranties of title, merchantability, fitness for a particular purpose, non infringement, or the absence of latent or other defects, accuracy, or the present or absence of errors, whether or not discoverable, all to the greatest extent permissible under applicable law.
    3. -
  3. Affirmer disclaims responsibility for clearing rights of other persons that may apply to the Work or any use thereof, including without limitation any person's Copyright and Related Rights in the Work. Further, Affirmer disclaims responsibility for obtaining any necessary consents, permissions or other rights required for any use of the Work.
    4. -
  4. Affirmer understands and acknowledges that Creative Commons is not a party to this document and has no duty or obligation with respect to this CC0 or use of the Work.
    5. -
diff --git a/src/main/assembly/dist/res/template/Appendix1.html b/src/main/assembly/dist/res/template/Appendix1.html new file mode 100644 index 0000000..ba179a2 --- /dev/null +++ b/src/main/assembly/dist/res/template/Appendix1.html @@ -0,0 +1,40 @@ +

Appendix 1 Dataset availability

+

The Dataset will be distributed and made available by the Depositary in the manner set out below. If the Dataset contains personal data within the meaning of the GDPR, with the exception of Bibliographical Data, the only access category permitted is Restricted Access. Metadata will always be made freely available.

+#if ($OpenAccess) +

Open Access

+

The files in the Dataset will be directly accessible to third parties. Third parties do not have to register with the Depositary. The Dataset will be placed in the public domain or made available under an Open Access Licence.

+

Using the following public-domain statement or Open Access Licence:

+

+ #if ($TermsLicenseUrl) + #if ($TermsLicense) + $TermsLicense : $TermsLicenseUrl + #else + $TermsLicenseUrl + #end + #else + #if ($TermsLicense) + $TermsLicense + #else + neither name nor URL for chosen license + #end + #end + #if ($Appendix3) + (see Appendix 3) + #end +

+#else +

Restricted Access

+

The files in the Dataset will be made available to Users by the Depositary only after the Depositor has given its express consent. Users will be obliged to comply with the DANS Licence. After receiving a permission request for access to the Dataset, the Depositor can approve the request, reject it or impose additional conditions of use on a User.

+

Using the following licence:

+

DANS Licence (see Appendix 2)

+#end +#if ($UnderEmbargo) +

Embargo

+

The files in the Dataset will not be made available to third parties for a limited period of up to two years from the commencement of this agreement. After this period has expired, the agreed access category and public-domain declaration or licence will automatically take effect. Extension of this period will only be possible for compelling reasons and in consultation with the Depositary.

+

Until the following date1:

+

$DateAvailable

+
+
+

1The end date will be a maximum of 2 years after the commencement of this agreement.

+
+#end diff --git a/src/main/assembly/dist/res/template/Appendix2.html b/src/main/assembly/dist/res/template/Appendix2.html new file mode 100644 index 0000000..8b41b55 --- /dev/null +++ b/src/main/assembly/dist/res/template/Appendix2.html @@ -0,0 +1,71 @@ +

Appendix 2 DANS Licence

+

DANS Licence

+

Effective from 9 January 2020

+

This licence sets out the conditions for using datasets to which the access categories “Open access for registered users” and “Restricted access” apply.

+

Anyone to whom DANS, on behalf of the holder of the rights to the dataset, makes one or more files of a dataset available (hereinafter referred to as the “User”), agrees to the following conditions. Acceptance of the conditions establishes an agreement between DANS and the User.

+
    +

  1. Responsible use

    +

    The User will act in accordance with the Netherlands Code of Conduct for Research Integrity, the GDPR and other applicable laws and regulations.

    +
    2. +

  2. Citing the dataset

    +

    The User will always cite the dataset in the research results they publish, in whatever form, when it has been used in the research.

    +

    This source reference will at least consist of:

    +
      +
    • The names and/or organisations of the producers of the dataset;
      • +
    • The year in which the dataset was produced;
      • +
    • The title of the dataset;
      • +
    • The name of organisation managing the archive in which the dataset is stored: DANS;
      • +
    • The persistent identifier of the dataset as a full URL.
      • +
    +

    For example:

    +

    +

    + Doorn, dr P.K.; Bommeljé, drs L.S.; Vroom, dr J.A.C.; Wijngaarden, drs H. van; +
    + Bommeljé, drs Y.B. (1990): The Aetolian Studies Project. +
    + DANS. https://doi.org/10.17026/dans-xxu-6utq +

    +
    3. +

  3. Distribution or disclosure of the dataset

    +

    The User shall respect all intellectual property rights to the dataset, such as copyrights, database and/or neighbouring rights.

    +

    For distribution or disclosure of the entire dataset or of substantial parts thereof, the User must first request permission from the holder of the rights to the dataset. This is the person(s) and/or institution(s) listed in the “Rights holder” metadata field of the dataset. If no holder is listed in this field, the User must contact the person(s) and/or organisation of the person(s) who produced the dataset.

    +
    4. +

  4. Statement when distributing or disclosing the dataset

    + +

    When distributing or disclosing the entire dataset or substantial parts of it, in the manner described in Article 2 of this DANS Licence and with the permission of the rights holder, the User shall, in addition to the acknowledgement referred to in Article 2, declare at all times:

    +
      +
    • the name of the dataset rights holder;
      • +
    • that this rights holder has granted permission for the distribution;
      • +
    • that further distribution by third parties is not permitted without the consent of the dataset rights holder.
      • +
    +
    5. +

  5. Publications

    +

    The User will inform DANS of the publications for which the dataset has been used. In this context, publications are defined as publications with an internationally recognized standard identification number, such as ISBN, ISSN or DOI. If a publication is available on the internet, the User will pass on the URL to DANS. If a publication is not available on the internet (via an URL), the User will pass on the source reference to DANS.

    +
    6. +

  6. Personal data

    +

    The User will always be responsible for the processing of personal data made available within the meaning of the GDPR and any other relevant privacy legislation, as well as for complying with any conditions set by the depositor.

    +
    7. +

  7. Liability for content

    +

    DANS shall in no way be liable for the contents or accompanying documentation of the dataset, including infringements of privacy rights within the meaning of the GDPR, unless in the event of intent or gross negligence on the part of DANS. The User is requested to inform DANS of any inaccuracies found as soon as possible after their discovery.

    +

    Neither DANS nor the depositor provide any guarantee that a dataset made available will meet the research objectives of the User. Neither DANS nor the depositor are liable for conclusions based on the dataset.

    +
    8. +

  8. Non-compliance with licence conditions

    +
      +
    1. If the licence conditions are not complied with, the use of the dataset must immediately be discontinued upon DANS’s first request. DANS reserves the right to inform the User's employer. In the event of unlawful use of personal data, DANS has the right to inform the Data Protection Authority as well. These measures are without prejudice to the authority of DANS to hold the User to account in court in the event of non-compliance or insufficient compliance with this licence.
      2. +
    2. If the licence conditions are not complied with, the User’s access to datasets other than that of the rights holder will be suspended until the issue has been resolved in consultation with the User, their employer and the rights holder (where applicable).
      3. +
    3. The User will indemnify DANS against all claims by third parties which are directly or indirectly related to the use of a dataset made available.
      4. +
    +
    9. +

  9. Compelling reasons

    +

    For compelling reasons, such as, but not limited to, an infringement of other people’s copyright or an infringement of the Code of Conduct for Research Integrity, DANS has the right to order the User to stop using the dataset.

    +
    10. +

  10. Changes to the agreement

    +

    DANS reserves the right to unilaterally change this agreement. In the event of substantial changes, DANS will inform the User, through EASY or by other means, before the new conditions take effect, so that the User has the opportunity to become aware of the changes. If the User does not accept the changes, the User must stop using the dataset(s) and delete any downloaded files. By continuing to use the dataset(s) after the changes have taken effect, the User accepts the updated conditions.

    +
    11. +

  11. Applicable law

    +
      +
    1. This licence is governed by Dutch law.
      2. +
    2. Disputes that cannot be resolved amicably will be submitted to the competent court in the Amsterdam district.
      3. +
    +
\ No newline at end of file diff --git a/src/main/assembly/dist/res/template/Body.html b/src/main/assembly/dist/res/template/Body.html index 3dd1f7c..73c133c 100644 --- a/src/main/assembly/dist/res/template/Body.html +++ b/src/main/assembly/dist/res/template/Body.html @@ -1,135 +1,161 @@ -

- Licence Agreement relating to
- Title of the study/research:
- “$Title” -

- -

The following parties are involved in this Licence Agreement:

-
    -
  1. The organisation or person authorised to transfer and deposit - the digital dataset(s), hereafter referred to as - the Depositor +

    Parties

    +
      +
    1. +

      + the organisation or person who is entitled to transfer and manage + the Dataset, hereinafter referred to as Depositor, +

      +
      +
      Name:
      $Name
      +
      Organisation:
      $Organisation
      +
      Address:
      $Address
      +
      Postal code:
      $PostalCode
      +
      Town:
      $City
      +
      Country:
      $Country
      +
      Telephone:
      $Telephone
      +
      Email:
      $Email
      +
      2. -
    2. The organisation that is authorised to archive and manage the - digital dataset(s), hereafter referred to as - the Repository +
    3. +

      + the organisation that is entitled to archive and manage + the Dataset, hereinafter referred to as Depositary, +

      +
      +
      Organisation:
      Data Archiving and Networked Services (DANS), on behalf of the Royal Netherlands Academy of Arts and Sciences (KNAW)
      +
      Represented by:
      Dr. P.K. Doorn, Director
      +
      Address:
      Anna van Saksenlaan 51
      +
      Postal code:
      2509 AB
      +
      Town:
      The Hague
      +
      Country:
      The Netherlands
      +
      Telephone:
      +31 (0)70 349 4450
      +
      E-mail:
      info@dans.knaw.nl
      +
    -

    - The Depositor is:
    - Name: $Name
    - Organisation: $Organisation
    - Postal address: $Address
    - Postal code: $PostalCode
    - Town/city: $City
    - Country: $Country
    - Tel.: $Telephone
    - E-mail: $Email
    + DANS is an institute of KNAW (Royal Netherlands Academy of Arts and Sciences) + and NWO (Dutch Research Council) with its registered office in The Hague.

    - -

    - The Repository is:
    - Organisation: DANS, Data Archiving and Networked Services, on behalf of KNAW (Royal Netherlands Academy of Arts and Sciences)
    - Represented by: Dr. P.K. Doorn, Director
    - Postal address: P.O. Box 93067
    - Postal code: 2509 AB
    - Town/city: The Hague
    - Country: The Netherlands
    - Tel.: +31 (0)70 349 4450
    - E-mail: info@dans.knaw.nl
    -

    - -

    DANS is an institute under the auspices of the Royal Netherlands Academy of Arts and Sciences (KNAW) which is also supported by the Netherlands Organisation for Scientific Research (NWO). DANS is based in The Hague

    - -
    - -

    This Licence Agreement is subject to the following provisions:

    - -

    1. Licence

    -
      -
    1. The Depositor grants the Repository a non-exclusive licence for digital data files, hereafter referred to as 'dataset'.
      2. -
    2. The Repository is authorised to include the dataset in its data archive. The Repository shall transfer the content of the dataset to an available carrier, through any method and in any form.
      3. -
    3. The Repository is authorised to make the dataset (or substantial parts thereof) available to third parties by means of on-line transmission. In addition, the Repository has the right, on the instruction of third parties or otherwise, to make a copy of the dataset or to grant third parties permission to download a copy.
      4. -
    - -

    2. The Depositor

    -
      -
    1. The Depositor declares that he is a holder of rights to the dataset, or the only holder of rights to the dataset, under the Databases act (Databankenwet) and where relevant the Copyright Act (Auteurswet) or otherwise, and/or is entitled to act in the present matter with the permission of other parties that hold rights.
      2. -
    2. The Depositor indemnifies the Repository against all claims made by other parties against the Repository with regard to the dataset, the transfer of the dataset, and the form and/or content of the dataset.
      3. -
    - -

    3. The Repository

    -
      -
    1. The Repository shall ensure, to the best of its ability and resources, that the deposited dataset is archived in a sustainable manner and remains legible and accessible.
      2. -
    2. The Repository shall, as far as possible, preserve the dataset unchanged in its original software format, taking account of current technology and the costs of implementation. The Repository has the right to modify the - format and/or functionality of the dataset if this is necessary in order to facilitate the digital sustainability, distribution or re-use of the dataset.
      3. -
    3. If the access categories “Restricted Access” or “Other Access”, as specified at the end of this Agreement, are selected, the Repository shall, to the best of its ability and resources, ensure that effective technical and other measures are in place to prevent unauthorised third parties from gaining access to and/or consulting the dataset or substantial parts thereof.
      4. -
    - -

    4. The dataset

    -
      -
    1. The dataset to which the licence relates is specified in the appendix to this Agreement. The appendix forms an integral part of this Agreement.
      2. -
    2. The Depositor declares that the dataset corresponds to the specification provided.
      3. -
    3. The Depositor declares that the dataset contains no data or other elements that are contrary to Dutch law.
      4. -
    4. The Depositor indemnifies the Repository against all claims by third parties relating to the content of the dataset.
      5. -
    5. The Depositor will supply the dataset by means of a method and medium deemed acceptable by the Repository.
      6. -
    - -

    5. Removal of datasets / changes to access conditions

    -
      -
    1. The Repository may decide not to make the dataset available to third parties for a temporary period or permanently as well as to remove the dataset from the archive wholly or in part. The Repository is entitled to do so if a Depositor has submitted a request stating the reasons for this action. In the case of not making the dataset available, the Repository shall retain the dataset in the data archive, but shall no longer allow third parties to access the dataset or substantial parts thereof.
      2. -
    2. If sufficient weighty grounds exist, the Repository has the right to remove the dataset from the archive wholly or in part, or to restrict or prevent access to the dataset on a temporary or permanent basis. The Repository shall inform the Depositor in such cases.
      3. -
    - -

    6. Availability to third parties:

    -
      -
    1. The Repository shall make the dataset available to third parties in accordance with the access conditions agreed with the Depositor: “Open Access”, “Open Access for Registered Users”, “Restricted Access” or “Other Access”. Access conditions may vary between parts of the dataset.
      2. -
    2. The Repository shall make the dataset available to third parties with whom they have reached agreement on the General Conditions for Use only if agreement has been reached with the Depositor on one of the following access categories: “Open Access for Registered Users”, “Restricted Access” or “Other Access”. Unless agreed otherwise with the Depositor, the use of datasets is subject to the General Conditions of Use laid down by the Repository. When the “Open Access” access category is agreed, the dataset shall be made available to third parties without the Depositor necessarily agreeing on the General Conditions for Use with said third parties.
      3. -
    3. If the access category “Restricted Access” has been agreed, the Repository shall make the dataset (or parts thereof) available only to the persons and/or organisations specified by the Depositor.
      4. -
    4. If a dataset (or parts thereof) to which the access categories “Restricted Access” or “Other Access” apply contain, as evidenced by the specification provided by the Depositor, personal data as referred to in the Personal Data Protection Act of The Netherlands (WBP; Wet Bescherming Persoonsgegevens, Act of 6 July 2000, Bulletin of Acts and Decrees 302, Article 9 paragraph 3 and Article 23 paragraph 2), the Repository shall make the dataset (or parts thereof) available only if this is permitted by law, which in any case should be taken to include making the dataset (or parts thereof) available for the purpose of historical, statistical or scientific research.
      5. -
    5. Notwithstanding the above, the Repository can make the dataset (or substantial parts thereof) available to third parties: -
        -
      • if the Repository is required to do so by legislation or regulations, a court decision, or by a regulatory or other institution
        • -
      • if this is necessary for the preservation of the dataset and/or the data archive
        • -
      • (to a similar institution) if the Repository ceases to exist and/or its activities in the field of data-archiving are terminated
        • -
      -
      6. -
    6. The Repository shall publish the documentation, hereafter referred to as metadata, as provided by the Depositor with the dataset, and make this freely available. Metadata is defined in this agreement as the content of all the fields that has been filled in with this dataset as standard under the tab “Description” in EASY.
      7. -
    7. The general information about the research and the metadata relating to the dataset, as defined in article 6.f, shall be included in the Repository’s databases and publications that are freely accessible to all persons.
      8. -
    - -

    7. Provisions relating to use by third parties

    +

    Considering that:

      -
    1. The Repository shall require third parties to whom the dataset (or substantial parts thereof) is made available to include in the research results a clear reference to the dataset from which data have been used. The reference must comply with the DANS General Conditions of Use. This obligation does not apply if the “Open Access” access category has been agreed on. In that case, the Repository will make every effort to inform third parties that they should include in their research results in whatever form, a clear acknowledgement of the source of the datasets from which data have been used.
      2. -
    2. The Repository shall oblige any party or parties to which it makes the dataset available to respect any copyright or database rights relating to the dataset.
      3. +
    3. The objective of the Depositary is to make research files permanently and digitally available;
      4. +
    4. The Depositary will only include and publish research files in its digital archival system if this can be done in a responsible manner;
      5. +
    5. The Depositor wishes to make use of the possibilities for digital archiving and accessibility of the deposited Dataset offered by the Depositary.
    - -

    8. Death of the Depositor or Discontinuance of the Depositor's organisation

    -

    From the time that the Depositor dies or in the event that the Depositor's organisation ceases to exist and no notice of termination in accordance with Article 10 has been given and if no legal successors, other parties that hold rights to the dataset or other persons mentioned in the field Rights Holder or in the field Creator of the metadata relating to the dataset, as defined in article 6.f, are known to DANS, the Repository shall be entitled to do whatever it considers reasonable in order to realise its objectives.

    - -

    9. Liability

    +

    Agree as follows:

      -
    1. The Repository accepts no liability in the event that all or part of a dataset is lost.
      2. -
    2. The Repository accepts no liability for any damage or losses resulting from acts or omissions by third parties to whom the Repository has made the dataset available.
      3. -
    - -

    10. Term, cancellation and termination of the Agreement

    -
      -
    1. This Agreement shall come into effect on the date on which the Repository publishes the dataset (hereafter the date of publication) and shall remain valid for an indefinite period. If the repository decides not to include the dataset in its data archive, this Agreement is cancelled. The Repository notifies the Depositor of publication or non-inclusion of the dataset in its data archive. Cancellation of this Agreement is subject to a period of notice of six months, and notice shall be given in writing. It is possible to change the agreed access category at any time during the term of the Agreement.
      2. -
    2. Notwithstanding point (a), this Agreement shall end when the dataset is removed from the data archive in accordance with Article 5 of this Agreement.
      3. -
    3. If the Repository ceases to exist or terminates its data-archiving activities, the Repository shall attempt to transfer the data files to a similar organisation that will continue the Agreement with the Depositor under similar conditions if possible.
      4. +
    4. +

      Deposit agreement

      +
        +
      1. The Depositor will grant the Depositary a non-exclusive licence to the deposited digital data and associated metadata.
        2. +
      2. The Depositary will receive the right to include the Dataset in its digital archival system. The Depositary will transfer the contents of the Dataset to a compatible carrier in a manner and format of its choice.
        3. +
      3. Subject to the terms of this deposit agreement, the Depositary will receive the right to make the Dataset, or substantial parts of it, available to third parties by means of electronic distribution. In addition, the Depositary will have the right to make a copy of the Dataset, whether or not on behalf of third parties, or allow third parties to download a copy.
        4. +
      4. The licence will be granted free of charge. The Depositary will not charge costs for making the Dataset available.
        5. +
      +
      5. +
    5. +

      Depositor

      +
        +
      1. The Depositor declares to be the sole entitled party with regard to the intellectual property rights to the Dataset pursuant to, but not limited to, the Database Act, the Copyright Act and other relevant legislation, and/or to act with the permission of the titleholder(s) or co-titleholder(s).
        2. +
      2. The Depositor will indemnify the Depositary against all claims that others may make against the Depositary with regard to the Dataset or the deposit thereof, its format and/or contents or its availability for further research.
        3. +
      3. The Depositor confirms that the Dataset does not contain any data or other elements which, in isolation or upon disclosure outside the context of scientific research, are inconsistent with the Dutch Penal Code or other relevant national or international legislation.
        4. +
      +
      6. +
    6. +

      Depositary

      +
        +
      1. The Depositary will, to the best of its ability and resources, permanently archive the Dataset, preserving its readability and accessibility.
        2. +
      2. The Depositary will archive the Dataset unaltered and in its original software format as far as possible, taking into account the technological state of the art and the cost of implementation. The Depositary will have the right to change the design and/or functionality of the Dataset in so far as it is necessary to ensure the digital preservation, distribution or reusability of the Dataset.
        3. +
      3. If the Dataset has been assigned to the “Restricted Access” category, the Depositary will, to the best of its ability and resources, ensure effective technical provisions to prevent unauthorized third parties from accessing and/or consulting substantial parts of the Dataset.
        4. +
      4. The Depositor will remain the controller of the Dataset within the meaning of the General Data Protection Regulation (GDPR) insofar as the Dataset contains personal data within the meaning of the GDPR. The Depositary will be a processor within the meaning of the GDPR. If the Dataset contains personal data, the Depositor and the Depositary will conclude a processing agreement, except in the case of bibliographical data which exclusively refer to personal data that are necessary for the accountability of the Dataset, such as its creator, rights holders and citations (hereinafter: “Bibliographical Data”).
        5. +
      +
      7. +
    7. +

      Dataset

      +
        +
      1. The Dataset will consist of all the files transferred by the Depositor and the metadata provided by the Depositor. Metadata is understood to mean the contents of all fields that must be completed in the archival system at the time of deposit in order to describe the Dataset.
        2. +
      2. The Depositor declares that the Dataset corresponds to the metadata provided by the Depositor.
        3. +
      3. The Dataset shall be compiled with due observance of the Netherlands Code of Conduct for Research Integrity, the GDPR and other applicable laws and regulations.
        4. +
      4. The metadata and file names shall not contain any personal data within the meaning of the GDPR. Only Bibliographical Data are allowed. It is explicitly forbidden to include personal data that are part of the deposited data, such as - but not limited to - research subjects, in the metadata and file names.
        5. +
      5. The Depositor will provide the files in a preferred format, as defined on the Depositary's website at the time of deposit. In the event that a format is not defined as a preferred format, the Depositor will contact the Depositary before delivery. A different file format may only be supplied with the written consent of the Depositary.
        6. +
      6. The Depositor will provide documentation with the Dataset that explains its creation, contents and any specific values (such as codes, characters and abbreviations), its structure (such as folder structures and relationships between files) and its actual use (such as that of software) to third parties.
        7. +
      7. The Depositor will make the Dataset available to the Depositary in a manner and through a medium that the Depositary deems suitable.
        8. +
      +
      8. +
    8. +

      Removing the Dataset and/or changing its accessibility

      +
        +
      1. The Depositor may submit a reasoned request to the Depositary to make the Dataset temporarily or permanently unavailable to third parties or to remove it entirely or partly from the archival system. The Depositary will assess the request in view of its objective. In cases where the Dataset contains personal data, the Depositary will assess the request with due observance of the GDPR.
        2. +
      2. If there are compelling reasons to do so, the Depositary has the right to remove the Dataset, or part of it, from the archival system, or to limit or exclude access to it temporarily or permanently. In such cases, the Depositary will reasonably inform the Depositor.
        3. +
      +
      9. +
    9. +

      Availability to third parties

      +
        +
      1. All or part of the Dataset files will be made available to third parties in accordance with one or more access categories agreed with the Depositor: “Open Access” or “Restricted Access”, possibly with the addition of an embargo (see Appendix 1). If Restricted Access is applied, part of the files may be made available directly to third parties, in consultation with the Depositary.
        2. +
      2. If Open Access is applied, the Depositor determines whether the files in the Dataset are placed in the public domain or whether a licence (hereinafter: “Open Access Licence”) shall apply (see Appendix 1).
        3. +
      3. If the Open Access category has been agreed with the Depositor, the Depositary will make the Dataset available directly to third parties, providing either the public domain statement or the agreed Open Access Licence.
        4. +
      4. The Open Access Licence provided for the Open Access category by the Depositor will apply to all files and their contents, to the extent that they are protected by copyright. The Depositary will make the Dataset as a whole, as well as its content or parts of its content, available under the conditions of the specified licence.
        5. +
      5. If the files in the Dataset, or parts of them, contain personal data within the meaning of the GDPR, the Restricted Access category will be applied exclusively and the files will not be made available directly to third parties.
        6. +
      6. If the Depositor and Depositary agree on Restricted Access, the following shall apply: +
          +
        1. The Depositary will make the files in the Dataset, or parts of them, exclusively available to users registered with the Depositary with whom they have agreed the DANS Licence (see Appendix 2) (hereinafter: “Users”). The Depositor has taken note of the DANS Licence and agrees to make the Dataset available in accordance with this licence. The Depositor is free to impose further conditions on Users prior to making the Dataset available.
          2. +
        2. In addition to the aforementioned provisions, the Depositary will only make the files in the Dataset, or parts of them, available to Users who have submitted a permission request for access to the Dataset and who have subsequently been granted such permission by the Depositor.
          3. +
        3. If a Dataset with Restricted Access includes files which the Depositor wishes to provide without the required permission as referred to under (b), the Depositary shall make the files directly available to third parties at the request of the Depositor, referring to the DANS Licence.
          4. +
        4. The Depositor will ensure the availability of a stable email address and check it regularly so that permission requests can be processed within a reasonable period of time.
          5. +
        5. The Depositary will facilitate consultations between the Depositor and the User who wants to use a Dataset, but cannot be held responsible for the Depositor’s decision whether or not to make the Dataset available, nor for any conditions under which this is done. +
          6. +
        +
        7. +
      7. Contrary to the above, the Depositary may make the Dataset or substantial parts of it available to third parties: +
          +
        1. if the Depositary is obliged to do so by virtue of laws and regulations, a judicial decision, or by a supervisory body;
          2. +
        2. if this is necessary for preserving the Dataset or the archival system;
          3. +
        3. if the Depositary ceases to exist and/or terminates its activities in the field of data archiving, or transfers its activities to a similar institution in compliance with Article 8.
          4. +
        +
        8. +
      8. The Depositary will make the metadata associated with the Dataset freely available.
        9. +
      9. The metadata associated with the Dataset will be included in the Depositary’s databases and publications, and will be accessible to everyone.
        10. +
      10. When a Dataset no longer contains any personal data within the meaning of the GDPR, the Depositor and the Depositary may decide to change its access category.
        11. +
      +
    10. +

      Using Datasets with Restricted Access

      +
        +
      1. The Depositary will oblige Users to whom it makes a Dataset or substantial parts of it available to include in their research results an unambiguous acknowledgement of the source, as specified in the DANS Licence, regarding the Dataset whose data has been used.
        2. +
      2. The Depositary will oblige Users to whom it makes the Dataset available to respect any copyright and database rights pertaining to the Dataset, as indicated by the Depositor.
        3. +
      3. In the event that the Depositary finds a violation, the Depositary will contact the Depositor and the User will be excluded from further use of Datasets until the issue has been resolved with the Depositor.
        4. +
      4. The Depositor will notify the Depositary as soon as possible of a failure to cite the source or an infringement of copyright or database rights as referred to in this article, to allow the Depositary to take the measures that it deems necessary, including, but not limited to, exclusion from further use of Datasets.
        5. +
      +
      11. +
    11. +

      Death of the Depositor or liquidation of its organisation

      +

      Upon the death of the Depositor or the liquidation or termination of its organisation, if no legal successors or co-entitled parties are known to the Depositary, the Depositary will be entitled to do whatever it deems reasonably necessary with a view to achieving its objective. In the event that the Dataset contains personal data within the meaning of the GDPR, the Depositary will request another organisation to act as the Depositor. This request will only be made to an organisation established in the Netherlands that, according to its statutes or by law, has scientific research, or the support or promotion of it, among its core tasks and/or objectives.

      +
      12. +
    12. +

      Liability

      +
        +
      1. Barring intent and/or gross negligence, the Depositor will indemnify the Depositary against all liability resulting from complete or partial loss of the Dataset.
        2. +
      2. Barring intent and/or gross negligence, the Depositor will indemnify the Depositary against all claims from third parties that may be brought against the Depositary as a result of the Dataset being lost in whole or in part, as well as against the costs that the Depositary will incur, for example for legal proceedings.
        3. +
      +
      13. +
    13. +

      Duration, cancellation, termination of the agreement

      +
        +
      1. This agreement takes effect on the date of the acceptance of this deposit agreement by the Depositor. The Depositary will then publish the Dataset as soon as possible.
        2. +
      2. This agreement will remain in effect for an indefinite period of time, unless: +
          +
        1. Either of the parties cancels the agreement in writing, subject to a notice period of six months;
          2. +
        2. Pursuant to Article 5 of this agreement, the Dataset has been permanently removed from the archival system;
          3. +
        3. Article 8 applies.
          4. +
        +
      3. Changing the access category or Open Access Licence is always permitted at the written request of the Depositor, with due observance of Article 6(5) of this agreement.
        4. +
      +
    14. +

      Applicable law

      +
        +
      1. This agreement is governed by Dutch law.
        2. +
      2. Disputes that cannot be resolved amicably will be submitted to the competent court in the Amsterdam district.
        3. +
      +
    - -

    11. Jurisdiction

    -

    DANS is entitled, but not obliged, to act independently against violations of the Copyright Act (Auteurswet) and/or any other intellectual property right of the holder(s) of rights to the dataset and/or the data from the dataset.

    - -

    12. Applicable law

    -

    Dutch law is applicable to this agreement.

    - -
    - -

    Access categories for datasets:

    -

    The Repository is permitted to distribute the dataset and make it available by means of one of the methods mentioned below and, if indicated below, making use of the additional option Embargo. Notwithstanding the above, the Repository, after consultation with and on request of the Depositor, is permitted to distribute parts of the dataset and make these available by means of one of the methods mentioned below being another than the one chosen for the dataset if that is clearly indicated by the Depositor.

    - -

    You have chosen:

    diff --git a/src/main/assembly/dist/res/template/FileTable.html b/src/main/assembly/dist/res/template/FileTable.html deleted file mode 100644 index 11e9885..0000000 --- a/src/main/assembly/dist/res/template/FileTable.html +++ /dev/null @@ -1,21 +0,0 @@ -

    Uploaded files:

    - - - - - - - #foreach ($file in $FileTable) - - - - - - #end -
    FileSHA-1 checksumAccessible to
    $file.FilePath$file.FileChecksum$file.FileAccessibleTo
    - -#if ($shouldLimitFiles) -

    Please note that only the first $limitFiles files are listed here in order to keep this license at - a reasonable size. For the full list of files, please contact DANS at - info@dans.knaw.nl.

    -#end diff --git a/src/main/assembly/dist/res/template/Header-sample.html b/src/main/assembly/dist/res/template/Header-sample.html index 2db5726..1b13225 100644 --- a/src/main/assembly/dist/res/template/Header-sample.html +++ b/src/main/assembly/dist/res/template/Header-sample.html @@ -1,5 +1,13 @@ -

    Sample - Licence Agreement

    +

    Draft Deposit Agreement

    - Persistent Identifier: will be assigned after submitting the dataset
    - Deposit date: $DateSubmitted + This deposit agreement relates to the following digital data and associated metadata + (hereinafter collectively referred to as “Dataset”):

    +
    +
    Title:
    +
    $Title
    +
    Persistent identifier:
    +
    will be assigned after submitting the dataset
    +
    Deposit date:
    +
    .....................
    +
    diff --git a/src/main/assembly/dist/res/template/Header.html b/src/main/assembly/dist/res/template/Header.html index 482ad43..3bba0c5 100644 --- a/src/main/assembly/dist/res/template/Header.html +++ b/src/main/assembly/dist/res/template/Header.html @@ -1,5 +1,13 @@ -

    Licence Agreement

    +

    Deposit Agreement

    - Persistent Identifier: $DansManagedDoi
    - Deposit date: $DateSubmitted + This deposit agreement relates to the following digital data and associated metadata + (hereinafter collectively referred to as “Dataset”):

    +
    +
    Title:
    +
    $Title
    +
    Persistent identifier:
    +
    $DansManagedDoi
    +
    Deposit date:
    +
    $DateSubmitted
    +
    diff --git a/src/main/assembly/dist/res/template/Table.html b/src/main/assembly/dist/res/template/Table.html deleted file mode 100644 index 67a7b0e..0000000 --- a/src/main/assembly/dist/res/template/Table.html +++ /dev/null @@ -1,9 +0,0 @@ -

    List of all metadata provided for this dataset at $CurrentDateAndTime

    - -#foreach ($row in $MetadataTable) - - - - -#end -
    $row.MetadataKey$row.MetadataValue
    diff --git a/src/main/assembly/dist/res/template/style.css b/src/main/assembly/dist/res/template/style.css new file mode 100644 index 0000000..a5dc580 --- /dev/null +++ b/src/main/assembly/dist/res/template/style.css @@ -0,0 +1,158 @@ + +@page { + /* the bottom gets 2.5cm extra for the footer */ + margin: 2.5cm 1.5cm 4cm 1.5cm; + + size: A4; + @top-left { + content: ""; + width: 100%; + height: 100%; + background: url(data:image/jpg;base64,$DansLogo) no-repeat 0 0; + background-position: left bottom; + background-size: 235px 90px; + } + @bottom-right { + vertical-align: top; + margin-right: 8px; + font: 11px "DejaVu Serif"; + content: counter(page) "/" counter(pages); + } +} +body { + font: 12px "DejaVu Serif"; + text-align: justify; +} +pre { + font: 10px "DejaVu Sans Mono"; +} +footer { + position: fixed; + width: 100%; + bottom: -9em; + font: 10px "DejaVu Serif"; + background: #FFF url(data:image/jpg;base64,$DrivenByData) no-repeat 0 0; + background-size: 212px 33px; + + /* 4% in a PDF matches the 16px in "footer p:first-child { margin-right" */ + background-position: 96% 100%; +} +@media screen { + body { + margin-bottom: 9em; + } + footer { + bottom: 0; + } +} +footer p:first-child { + float: right; + + /* box model of appears to have a margin of 8px, + * the footer is 8px larger than the body with its margin + * which matches 4% for "footer { background-position" + */ + margin-right: 16px; + + margin-top: 2.5em; + text-align: right; + background-color: #FFF; +} +.choice, .choice-header { + padding-left: 5em; +} +.error { + color: red; + font-size: 150%; + font-weight: bold; +} +.choice-header { + /* NB: underline dances with by WeasyPrintPdfGenerator */ + text-decoration: underline; +} + +/* dataset, parties on front page. */ + +dt, dd { + min-height: 1em; /* avoid the need for   in an empty
    (part 1) */ +} +dt { + clear: left; /* avoid the need for   in an empty
    (part 2) */ + float: left; /* put it in the margin of
    */ + width: 10.5em; /* same as margin for
    */ +} +dd { + margin-left: 10.5em; /* place holder for the dt */ +} +li dt { + width: 9em; /* subtract padding-left of "body>ol" */ +} +li dd { + margin-left: 9em; +} + +/* first level numbered list, needs explicit counter for second level */ + +body>ol { + list-style-type: none; + counter-reset: level1; + padding-left: 1.5em; +} +body>ol>li { + counter-increment: level1; +} +body>ol>li>p:first-child { + font-weight: bold; + display: inline-block; +} +body>ol>li>p:first-child:before { + content: counters(level1, ".") ". "; + margin-left: -1.5em; + width: 1.5em; +} + +/* second level prefixed with first level */ + +.body>li>ol { + list-style-type: none; + counter-reset: level2; + padding-left: 3.5em; +} + +.body>li>ol>li { + counter-increment: level2; +} + +.body>li>ol>li:before { + content: counters(level1, ".") "." counters(level2, ".") "."; + display: inline-block; + margin-left: -3.5em; + width: 3.5em; +} + +/* lower-alpha numbering for different levels in different sections */ + +.body>li>ol>li>ol, +.appendix2>li>ol { + list-style-type: lower-alpha; + padding-left: 1.5em; +} + +/* unordered at any level */ + +ul { + list-style-type: disc; + padding-left: 1.8em; +} + +/* footnote */ + +.footnote { + padding-top: 2em; +} +hr { + color: #000; + height: 0; + width: 50%; + margin-left: 0; +} \ No newline at end of file diff --git a/src/main/scala/nl/knaw/dans/easy/agreement/AgreementCreator.scala b/src/main/scala/nl/knaw/dans/easy/agreement/AgreementCreator.scala index 90d1118..9abe2ee 100644 --- a/src/main/scala/nl/knaw/dans/easy/agreement/AgreementCreator.scala +++ b/src/main/scala/nl/knaw/dans/easy/agreement/AgreementCreator.scala @@ -15,13 +15,12 @@ */ package nl.knaw.dans.easy.agreement -import java.io.{ByteArrayInputStream, ByteArrayOutputStream, OutputStream} +import java.io.{ ByteArrayInputStream, ByteArrayOutputStream, OutputStream } import nl.knaw.dans.easy.agreement.internal._ import nl.knaw.dans.lib.logging.DebugEnhancedLogging import nl.knaw.dans.pf.language.emd.EasyMetadata -import rx.lang.scala.Observable -import rx.lang.scala.TryToObservable +import rx.lang.scala.{ Observable, TryToObservable } import scala.util.Try @@ -34,7 +33,7 @@ class AgreementCreator(placeholderMapper: PlaceholderMapper, trace(dataset, outputStream) resource.managed(new ByteArrayOutputStream()) .map(templateOut => { - logger.info(s"""creating the agreement for dataset "${dataset.datasetID}"""") + logger.info(s"""creating the agreement for dataset "${ dataset.datasetID }"""") for { placeholders <- placeholderMapper.datasetToPlaceholderMap(dataset.validate) _ <- templateResolver.createTemplate(templateOut, placeholders) @@ -64,7 +63,7 @@ class AgreementCreatorWithDatasetLoader(datasetLoader: DatasetLoader, // used in Easy-Ingest-Flow def createAgreement(emd: EasyMetadata, depositorID: DepositorID, files: Seq[FileItem]) (outputStream: OutputStream): Observable[Nothing] = { - datasetLoader.getDataset(parameters.datasetID, emd, depositorID, files, parameters.fileLimit) + datasetLoader.getDataset(parameters.datasetID, emd, depositorID) .flatMap(createAgreement(_)(outputStream).toObservable) .filter(_ => false) .asInstanceOf[Observable[Nothing]] diff --git a/src/main/scala/nl/knaw/dans/easy/agreement/AgreementCreatorApp.scala b/src/main/scala/nl/knaw/dans/easy/agreement/AgreementCreatorApp.scala index 15af853..2e7c0ff 100644 --- a/src/main/scala/nl/knaw/dans/easy/agreement/AgreementCreatorApp.scala +++ b/src/main/scala/nl/knaw/dans/easy/agreement/AgreementCreatorApp.scala @@ -16,9 +16,9 @@ package nl.knaw.dans.easy.agreement import java.io.File -import javax.naming.Context import com.yourmediashelf.fedora.client.{ FedoraClient, FedoraCredentials } +import javax.naming.Context import rx.schedulers.Schedulers class AgreementCreatorApp(configuration: Configuration) extends AutoCloseable { @@ -28,11 +28,6 @@ class AgreementCreatorApp(configuration: Configuration) extends AutoCloseable { configuration.properties.getString("fcrepo.url"), configuration.properties.getString("fcrepo.user"), configuration.properties.getString("fcrepo.password"))) - val fsrdb: (String, String, String) = ( - configuration.properties.getString("fsrdb.db-connection-url"), - configuration.properties.getString("fsrdb.db-connection-username"), - configuration.properties.getString("fsrdb.db-connection-password")) - val fileLimit: Int = configuration.properties.getInt("agreement.fileLimit") val ldapEnv: LdapEnv = new LdapEnv { put(Context.PROVIDER_URL, configuration.properties.getString("auth.ldap.url")) put(Context.SECURITY_AUTHENTICATION, "simple") diff --git a/src/main/scala/nl/knaw/dans/easy/agreement/AgreementCreatorServlet.scala b/src/main/scala/nl/knaw/dans/easy/agreement/AgreementCreatorServlet.scala index 3db5c7c..3296d12 100644 --- a/src/main/scala/nl/knaw/dans/easy/agreement/AgreementCreatorServlet.scala +++ b/src/main/scala/nl/knaw/dans/easy/agreement/AgreementCreatorServlet.scala @@ -56,7 +56,7 @@ class AgreementCreatorServlet(app: AgreementCreatorApp) extends ScalatraServlet } private def validateDatasetIdExistsInFedora(pars: Params): Try[Unit] = { - logger.info(s"check if dataset ${pars.datasetID} exists") + logger.info(s"check if dataset ${ pars.datasetID } exists") pars.fedora.datasetIdExists(pars.datasetID).flatMap { case true => Success(()) case false => Failure(new NoSuchElementException(s"DatasetId ${ pars.datasetID } does not exist")) @@ -70,8 +70,6 @@ class AgreementCreatorServlet(app: AgreementCreatorApp) extends ScalatraServlet isSample = params.get("sample").fold(false)(_.toBoolean), fedoraClient = app.fedoraClient, ldapEnv = app.ldapEnv, - fsrdb = app.fsrdb, - fileLimit = app.fileLimit, ) } diff --git a/src/main/scala/nl/knaw/dans/easy/agreement/Command.scala b/src/main/scala/nl/knaw/dans/easy/agreement/Command.scala index a988b1a..74b2412 100644 --- a/src/main/scala/nl/knaw/dans/easy/agreement/Command.scala +++ b/src/main/scala/nl/knaw/dans/easy/agreement/Command.scala @@ -55,7 +55,7 @@ object Command extends App with DebugEnhancedLogging { _ = logger.debug(s"Output will be written to ${ outputFile.getAbsolutePath }") _ <- validateDatasetIdExists(params) _ = createAgreement(outputFile, params) - _ = params.close() + _ = params.close() // TODO not reached when validation fails } yield () } .recoverWith { case t: Throwable => Failure(new Exception(s"Could not create agreement for ${ commandLine.datasetID() }: ${ t.getMessage }")) } @@ -72,14 +72,16 @@ object Command extends App with DebugEnhancedLogging { private def createAgreement(outputFile: File, params: internal.Parameters): Unit = { new FileOutputStream(outputFile) .usedIn(AgreementCreator(params).createAgreement) - .doOnCompleted { - logger.info(s"agreement saved at ${ outputFile.getAbsolutePath }") - } + .doOnCompleted(logger.info(s"agreement saved at ${ outputFile.getAbsolutePath }")) .toBlocking .subscribe( _ => {}, - e => logger.error("An error was caught in main:", e), - () => logger.debug("completed")) + e => { + logger.error("An error was caught in main:", e) + throw e + }, + () => logger.debug("completed") + ) } private def createParameters(app: AgreementCreatorApp): Try[internal.Parameters] = Try { @@ -89,8 +91,7 @@ object Command extends App with DebugEnhancedLogging { isSample = commandLine.isSample(), fedoraClient = app.fedoraClient, ldapEnv = app.ldapEnv, - fsrdb = app.fsrdb, - fileLimit = app.fileLimit) + ) } private def runAsService(app: AgreementCreatorApp): Try[FeedBackMessage] = Try { diff --git a/src/main/scala/nl/knaw/dans/easy/agreement/Parameters.scala b/src/main/scala/nl/knaw/dans/easy/agreement/Parameters.scala index 4b13b68..d2f22f8 100644 --- a/src/main/scala/nl/knaw/dans/easy/agreement/Parameters.scala +++ b/src/main/scala/nl/knaw/dans/easy/agreement/Parameters.scala @@ -18,12 +18,12 @@ package nl.knaw.dans.easy.agreement import java.io.File import com.yourmediashelf.fedora.client.FedoraClient -import nl.knaw.dans.easy.agreement.internal.{BaseParameters => InternalBaseParams, Parameters => InternalParams} +import nl.knaw.dans.easy.agreement.internal.{ BaseParameters => InternalBaseParams, Parameters => InternalParams } object BaseParameters { - def apply(templateResourceDir: File, datasetID: DatasetID, isSample: Boolean, fileLimit: Int): InternalBaseParams = { - new internal.BaseParameters(templateResourceDir, datasetID, isSample, fileLimit) + def apply(templateResourceDir: File, datasetID: DatasetID, isSample: Boolean): InternalBaseParams = { + new internal.BaseParameters(templateResourceDir, datasetID, isSample) } } @@ -32,10 +32,9 @@ object Parameters { def apply(templateResourceDir: File, datasetID: DatasetID, isSample: Boolean, - fileLimit: Int, fedoraClient: FedoraClient, ldapEnv: LdapEnv, - fsrdb: (String, String, String)): InternalParams = { - new internal.Parameters(templateResourceDir, datasetID, isSample, fileLimit, fedoraClient, ldapEnv, fsrdb) + ): InternalParams = { + new internal.Parameters(templateResourceDir, datasetID, isSample, fedoraClient, ldapEnv) } } diff --git a/src/main/scala/nl/knaw/dans/easy/agreement/internal/DatasetLoader.scala b/src/main/scala/nl/knaw/dans/easy/agreement/internal/DatasetLoader.scala index bea74dc..6c1cd6e 100644 --- a/src/main/scala/nl/knaw/dans/easy/agreement/internal/DatasetLoader.scala +++ b/src/main/scala/nl/knaw/dans/easy/agreement/internal/DatasetLoader.scala @@ -15,31 +15,28 @@ */ package nl.knaw.dans.easy.agreement.internal -import java.sql.SQLException import javax.naming.directory.Attributes - -import nl.knaw.dans.easy.agreement.{ DatasetID, DepositorID, FileAccessRight, FileItem } +import nl.knaw.dans.easy.agreement.{ DatasetID, DepositorID } import nl.knaw.dans.pf.language.emd.binding.EmdUnmarshaller -import nl.knaw.dans.pf.language.emd.{ EasyMetadata, EasyMetadataImpl, EmdAudience } +import nl.knaw.dans.pf.language.emd.{ EasyMetadata, EasyMetadataImpl } +import rx.lang.scala.Observable import rx.lang.scala.schedulers.IOScheduler -import rx.lang.scala.{ Observable, ObservableExtensions } -import scala.collection.JavaConverters._ import scala.language.postfixOps import scala.util.control.NonFatal /** - * Data class for an Easy User. Notice that some fields are mandatory and cannot be null! - * - * @param name the user's name (mandatory!) - * @param organization the user's organisation - * @param address the user's address (mandatory!) - * @param postalCode the user's zipcode (mandatory!) - * @param city the user's city (mandatory!) - * @param country the user's country - * @param telephone the user's telephone - * @param email the user's email (mandatory!) - */ + * Data class for an Easy User. Notice that some fields are mandatory and cannot be null! + * + * @param name the user's name (mandatory!) + * @param organization the user's organisation + * @param address the user's address (mandatory!) + * @param postalCode the user's zipcode (mandatory!) + * @param city the user's city (mandatory!) + * @param country the user's country + * @param telephone the user's telephone + * @param email the user's email (mandatory!) + */ case class EasyUser(name: String, organization: String, address: String, @@ -56,12 +53,7 @@ case class EasyUser(name: String, require(email != null, "'email' must be defined") } -case class Dataset(datasetID: DatasetID, - emd: EasyMetadata, - easyUser: EasyUser, - audiences: Seq[AudienceTitle], - fileItems: Seq[FileItem], - filesLimited: Boolean) { +case class Dataset(datasetID: DatasetID, emd: EasyMetadata, easyUser: EasyUser) { require(datasetID != null, "'datasetID' must be defined") @@ -71,84 +63,37 @@ case class Dataset(datasetID: DatasetID, trait DatasetLoader { /** - * Queries the audience title from Fedora given an audience identifiers - * - * @param audienceID the identifier of the audience - * @return The audidence title corresponding to the `audienceID` - */ - def getAudience(audienceID: AudienceID): Observable[AudienceTitle] - - /** - * Queries the audience titles from Fedora for the audiences in `EmdAudience`. - * @param audience the audience object with the identifiers - * @return the titles of the audiences that correspond to the identifiers in `EmdAudience` - */ - def getAudiences(audience: EmdAudience): Observable[AudienceTitle] = { - audience.getValues.asScala.toObservable.flatMap(getAudience) - } - - /** - * Create a `Dataset` based on the given `datasetID` - * - * @param datasetID the identifier of the dataset - * @return the dataset corresponding to `datasetID` - */ + * Create a `Dataset` based on the given `datasetID` + * + * @param datasetID the identifier of the dataset + * @return the dataset corresponding to `datasetID` + */ def getDatasetById(datasetID: DatasetID): Observable[Dataset] /** - * Create a `Dataset` based on the given `datasetID`, `emd`, `depositorID` and `files`, while - * querying for the audience titles and the depositor data. - * - * @param datasetID the identifier of the dataset - * @param emd the `EasyMetadata` of the dataset - * @param depositorID the depositor's identifier - * @param files the files belonging to the dataset - * @return the dataset corresponding to `datasetID` - */ - def getDataset(datasetID: DatasetID, emd: EasyMetadata, depositorID: DepositorID, files: Seq[FileItem], fileLimit: Int): Observable[Dataset] = { - val audiences = getAudiences(emd.getEmdAudience).toSeq - val easyUser = getUserById(depositorID) - val filesWereLimited = countFiles(datasetID).map(fileLimit <) - - easyUser.combineLatest(audiences) - .combineLatestWith(filesWereLimited) { - case ((user, auds), limited) => Dataset(datasetID, emd, user, auds, files, limited) - } + * Create a `Dataset` based on the given `datasetID`, `emd` and `depositorID`, while querying for the depositor data. + * + * @param datasetID the identifier of the dataset + * @param emd the `EasyMetadata` of the dataset + * @param depositorID the depositor's identifier + * @return the dataset corresponding to `datasetID` + */ + def getDataset(datasetID: DatasetID, emd: EasyMetadata, depositorID: DepositorID): Observable[Dataset] = { + getUserById(depositorID) + .map(Dataset(datasetID, emd, _)) } /** - * Returns all files corresponding to the dataset with identifier `datasetID` - * - * @param datasetID the identifier of the dataset - * @return the files corresponding to the dataset with identifier `datasetID` - */ - def getFilesInDataset(datasetID: DatasetID): Observable[FileItem] - - /** - * Return the number of files corresponding to the dataset with identifier `datasetID` + * Queries the user data given a `depositorID` * - * @param datasetID the identifier of the dataset - * @return the number of files corresponding to the dataset with identifier `datasetID` + * @param depositorID the identifier of the user + * @return the user data corresponding to the `depositorID` */ - def countFiles(datasetID: DatasetID): Observable[Int] - - /** - * Queries the user data given a `depositorID` - * - * @param depositorID the identifier of the user - * @return the user data corresponding to the `depositorID` - */ def getUserById(depositorID: DepositorID): Observable[EasyUser] } case class DatasetLoaderImpl()(implicit parameters: DatabaseParameters) extends DatasetLoader { - def getAudience(audienceID: AudienceID): Observable[String] = { - parameters.fedora.getDC(audienceID) - .map(resource.managed(_).acquireAndGet(_.loadXML \\ "title" text)) - .subscribeOn(IOScheduler()) - } - def getDatasetById(datasetID: DatasetID): Observable[Dataset] = { val emdObs = parameters.fedora.getEMD(datasetID) .map(resource.managed(_).acquireAndGet(new EmdUnmarshaller(classOf[EasyMetadataImpl]).unmarshal)) @@ -158,75 +103,10 @@ case class DatasetLoaderImpl()(implicit parameters: DatabaseParameters) extends .subscribeOn(IOScheduler()) .flatMap(getUserById(_).subscribeOn(IOScheduler())) - // publish because emd is used in multiple places here - emdObs.publish(emd => { - emd.combineLatestWith(depositorObs) { - (emdValue, depositorValue) => (audiences: Seq[AudienceTitle]) => (files: Seq[FileItem]) => (limited: Boolean) => - Dataset(datasetID, emdValue, depositorValue, audiences, files, limited) - } - .combineLatestWith(emd.map(_.getEmdAudience).flatMap(getAudiences).toSeq)(_(_)) - .combineLatestWith(getFilesInDataset(datasetID).toSeq)(_(_)) - .combineLatestWith(countFiles(datasetID).map(parameters.fileLimit <))(_(_)) - .single - .onErrorResumeNext { - case e: IllegalArgumentException => Observable.error(MultipleDatasetsFoundException(datasetID, e)) - case e: NoSuchElementException => Observable.error(NoDatasetFoundException(datasetID, e)) - case NonFatal(e) => Observable.error(e) - } - }) - } - - def getFilesInDataset(datasetID: DatasetID): Observable[FileItem] = { - Class.forName("org.postgresql.Driver") - val query = "SELECT pid, path, sha1checksum, accessible_to FROM easy_files WHERE dataset_sid = ? ORDER BY pid LIMIT ?;" - Observable.using(parameters.fsrdb.prepareStatement(query))( - prepStatement => { - prepStatement.setString(1, datasetID) - prepStatement.setInt(2, parameters.fileLimit) - - Observable.using(prepStatement.executeQuery())( - resultSet => Observable.defer(Observable.just(resultSet.next())) - .repeat - .takeWhile(b => b) - .map(_ => { - val pid = resultSet.getString("pid") - val path = resultSet.getString("path") - val sha1checksum = resultSet.getString("sha1checksum") - val accessibleTo = FileAccessRight.valueOf(resultSet.getString("accessible_to")) - .getOrElse(throw new IllegalArgumentException(s"illegal value for accessibleTo in file: $pid")) - - FileItem(path, accessibleTo, if (sha1checksum == "null") None else Some(sha1checksum)) - }), - _.close(), - disposeEagerly = true - ) - }, - _.close(), - disposeEagerly = true - ) - } - - def countFiles(datasetID: DatasetID): Observable[Int] = { - Class.forName("org.postgresql.Driver") - val query = "SELECT COUNT(pid) FROM easy_files WHERE dataset_sid = ?;" - Observable.using(parameters.fsrdb.prepareStatement(query))( - prepStatement => { - prepStatement.setString(1, datasetID) - - Observable.using(prepStatement.executeQuery())( - resultSet => { - if (resultSet.next()) - Observable.just(resultSet.getInt("count")) - else - Observable.error(new SQLException(s"unable to count the number of files in dataset $datasetID")) - }, - _.close(), - disposeEagerly = true - ) - }, - _.close(), - disposeEagerly = true - ) + emdObs.combineLatestWith(depositorObs) { + (emdValue, depositorValue) => + Dataset(datasetID, emdValue, depositorValue) + } } private def get(attrID: String)(implicit attrs: Attributes): Option[String] = { diff --git a/src/main/scala/nl/knaw/dans/easy/agreement/internal/Fedora.scala b/src/main/scala/nl/knaw/dans/easy/agreement/internal/Fedora.scala index 2731d9e..5ab20bb 100644 --- a/src/main/scala/nl/knaw/dans/easy/agreement/internal/Fedora.scala +++ b/src/main/scala/nl/knaw/dans/easy/agreement/internal/Fedora.scala @@ -18,7 +18,6 @@ package nl.knaw.dans.easy.agreement.internal import java.io.InputStream import com.yourmediashelf.fedora.client.{ FedoraClient, FedoraClientException, FedoraCredentials } -import com.yourmediashelf.fedora.generated.management.DatastreamProfile import nl.knaw.dans.easy.agreement.DatasetID import rx.lang.scala.Observable @@ -34,14 +33,6 @@ trait Fedora { */ def getAMD(pid: DatasetID): Observable[InputStream] - /** - * Queries Fedora for the DC datastream dissemination xml of the dataset with `identifier = pid`. - * - * @param pid identifier of the dataset to be queried - * @return the resulting `InputStream` wrapped in an `Observable` - */ - def getDC(pid: DatasetID): Observable[InputStream] - /** * Queries Fedora for the EMD datastream dissemination xml of the dataset with `identifier = pid`. * @@ -50,22 +41,6 @@ trait Fedora { */ def getEMD(pid: DatasetID): Observable[InputStream] - /** - * Queries Fedora for the FILE_METADATA datastream dissemination xml of the dataset with `identifier = pid`. - * - * @param pid identifier of the dataset to be queried - * @return the resulting `InputStream` wrapped in an `Observable` - */ - def getFileMetadata(pid: FileID): Observable[InputStream] - - /** - * Queries Fedora for the EASY_FILE datastream of the dataset with `identifier = pid`. - * - * @param pid identifier of the dataset to be queried - * @return the resulting `DatastreamProfile` wrapped in an `Observable` - */ - def getFile(pid: FileID): Observable[DatastreamProfile] - /** * Queries whether the provided datasetID exists in Fedora * @@ -89,18 +64,8 @@ case class FedoraImpl(client: FedoraClient) extends Fedora { def getAMD(pid: DatasetID): Observable[InputStream] = query(pid, "AMD").single - def getDC(pid: DatasetID): Observable[InputStream] = query(pid, "DC").single - def getEMD(pid: DatasetID): Observable[InputStream] = query(pid, "EMD").single - def getFileMetadata(pid: FileID): Observable[InputStream] = query(pid, "EASY_FILE_METADATA").single - - def getFile(pid: FileID): Observable[DatastreamProfile] = { - Observable.just(FedoraClient.getDatastream(pid, "EASY_FILE") - .execute(client) - .getDatastreamProfile) - } - override def datasetIdExists(datasetID: DatasetID): Try[Boolean] = Try { FedoraClient.getObjectXML(datasetID) .execute(client) diff --git a/src/main/scala/nl/knaw/dans/easy/agreement/internal/KeywordMapping.scala b/src/main/scala/nl/knaw/dans/easy/agreement/internal/KeywordMapping.scala index c076010..6c6ba75 100644 --- a/src/main/scala/nl/knaw/dans/easy/agreement/internal/KeywordMapping.scala +++ b/src/main/scala/nl/knaw/dans/easy/agreement/internal/KeywordMapping.scala @@ -18,8 +18,9 @@ package nl.knaw.dans.easy.agreement.internal // @formatter:off trait KeywordMapping { val keyword: String } -// logo in header (for 64bit encoding) +// logos in header and footer (for 64bit encoding) case object DansLogo extends KeywordMapping { val keyword = "DansLogo" } +case object DrivenByData extends KeywordMapping { val keyword = "DrivenByData" } // footer text case object FooterText extends KeywordMapping { val keyword = "FooterText" } @@ -43,30 +44,10 @@ case object DepositorEmail extends KeywordMapping { val keyword = // access rights case object OpenAccess extends KeywordMapping { val keyword = "OpenAccess" } -case object OpenAccessForRegisteredUsers extends KeywordMapping { val keyword = "OpenAccessForRegisteredUsers" } -case object OtherAccess extends KeywordMapping { val keyword = "OtherAccess" } -case object RestrictGroup extends KeywordMapping { val keyword = "RestrictGroup" } -case object RestrictRequest extends KeywordMapping { val keyword = "RestrictRequest" } - -// embargo case object UnderEmbargo extends KeywordMapping { val keyword = "UnderEmbargo" } case object DateAvailable extends KeywordMapping { val keyword = "DateAvailable" } case object CurrentDateAndTime extends KeywordMapping { val keyword = "CurrentDateAndTime" } - -// metadata table -case object MetadataTable extends KeywordMapping { val keyword = "MetadataTable" } -case object MetadataKey extends KeywordMapping { val keyword = "MetadataKey" } -case object MetadataValue extends KeywordMapping { val keyword = "MetadataValue" } - -// file table -case object HasFiles extends KeywordMapping { val keyword = "HasFiles" } -case object FileTable extends KeywordMapping { val keyword = "FileTable" } -case object FilePath extends KeywordMapping { val keyword = "FilePath" } -case object FileChecksum extends KeywordMapping { val keyword = "FileChecksum" } -case object FileAccessibleTo extends KeywordMapping { val keyword = "FileAccessibleTo" } - -// file limit -case object LimitFiles extends KeywordMapping { val keyword = "limitFiles" } -case object ShouldLimitFiles extends KeywordMapping { val keyword = "shouldLimitFiles" } -// @formatter:on +case object TermsLicense extends KeywordMapping { val keyword = "TermsLicense" } +case object TermsLicenseUrl extends KeywordMapping { val keyword = "TermsLicenseUrl" } +case object Appendix3 extends KeywordMapping { val keyword = "Appendix3" }// @formatter:on diff --git a/src/main/scala/nl/knaw/dans/easy/agreement/internal/Parameters.scala b/src/main/scala/nl/knaw/dans/easy/agreement/internal/Parameters.scala index 99650bc..2a66b2c 100644 --- a/src/main/scala/nl/knaw/dans/easy/agreement/internal/Parameters.scala +++ b/src/main/scala/nl/knaw/dans/easy/agreement/internal/Parameters.scala @@ -16,43 +16,57 @@ package nl.knaw.dans.easy.agreement.internal import java.io.File -import java.sql.{ Connection, DriverManager } -import javax.naming.ldap.InitialLdapContext import com.yourmediashelf.fedora.client.FedoraClient +import javax.naming.ldap.InitialLdapContext import nl.knaw.dans.easy.agreement.{ DatasetID, LdapEnv } +import org.apache.commons.configuration.PropertiesConfiguration + +import scala.collection.JavaConverters._ +import scala.util.Try // this class needs to be in a separate file rather than in package.scala because of interop with // java business layer. class BaseParameters(val templateResourceDir: File, val datasetID: DatasetID, val isSample: Boolean, - val fileLimit: Int) + ) { + private val licenseUrlPrefixRegExp = "https?://(www.)?" + val licencesMap: Map[String, String] = Try { + val file = new File(templateResourceDir, "/template/licenses/licenses.properties") + val licenses = new PropertiesConfiguration(file) + licenses.getKeys.asScala.map(key => + key.replaceAll(licenseUrlPrefixRegExp, "") -> s"/licenses/${ licenses.getString(key) }" + ).toMap + }.getOrElse(Map.empty) + + def licenseLegalResource(url: String): String = { + licencesMap.getOrElse( + url.replaceAll(licenseUrlPrefixRegExp, ""), + throw new IllegalArgumentException(s"No legal text found for $url") + ) + } +} trait DatabaseParameters { val fedora: Fedora val ldap: Ldap - val fsrdb: Connection - val fileLimit: Int } case class Parameters(override val templateResourceDir: File, override val datasetID: DatasetID, override val isSample: Boolean, - override val fileLimit: Int, fedora: Fedora, - ldap: Ldap, - fsrdb: Connection) - extends BaseParameters(templateResourceDir, datasetID, isSample, fileLimit) with DatabaseParameters with AutoCloseable { + ldap: Ldap) + extends BaseParameters(templateResourceDir, datasetID, isSample) with DatabaseParameters with AutoCloseable { - def this(templateResourceDir: File, datasetID: DatasetID, isSample: Boolean, fileLimit: Int, fedoraClient: FedoraClient, ldapEnv: LdapEnv, fsrdb: (String, String, String)) = { - this(templateResourceDir, datasetID, isSample, fileLimit, FedoraImpl(fedoraClient), LdapImpl(new InitialLdapContext(ldapEnv, null)), DriverManager.getConnection(fsrdb._1, fsrdb._2, fsrdb._3)) + def this(templateResourceDir: File, datasetID: DatasetID, isSample: Boolean, fedoraClient: FedoraClient, ldapEnv: LdapEnv) = { + this(templateResourceDir, datasetID, isSample, FedoraImpl(fedoraClient), LdapImpl(new InitialLdapContext(ldapEnv, null))) } - override def close(): Unit = { - fsrdb.close() - ldap.close() - } + override def close(): Unit = { + ldap.close() + } override def toString: String = super.toString } diff --git a/src/main/scala/nl/knaw/dans/easy/agreement/internal/PlaceholderMapper.scala b/src/main/scala/nl/knaw/dans/easy/agreement/internal/PlaceholderMapper.scala index 0b4bb49..6334e98 100644 --- a/src/main/scala/nl/knaw/dans/easy/agreement/internal/PlaceholderMapper.scala +++ b/src/main/scala/nl/knaw/dans/easy/agreement/internal/PlaceholderMapper.scala @@ -21,20 +21,18 @@ import java.{ util => ju } import nl.knaw.dans.common.lang.dataset.AccessCategory import nl.knaw.dans.common.lang.dataset.AccessCategory._ +import nl.knaw.dans.easy.agreement.FileAccessRight import nl.knaw.dans.easy.agreement.FileAccessRight._ -import nl.knaw.dans.easy.agreement.{ DatasetID, FileAccessRight, FileItem } import nl.knaw.dans.lib.error.TryExtensions import nl.knaw.dans.lib.logging.DebugEnhancedLogging -import nl.knaw.dans.lib.string.StringExtensions -import nl.knaw.dans.pf.language.emd.types.Spatial.{ Box, Point } import nl.knaw.dans.pf.language.emd.types._ -import nl.knaw.dans.pf.language.emd.{ EasyMetadata, EmdDate, Term } +import nl.knaw.dans.pf.language.emd.{ EasyMetadata, EmdDate } import org.apache.commons.codec.binary.Base64 import org.apache.commons.io.FileUtils import org.joda.time.DateTime import scala.collection.JavaConverters._ -import scala.collection.{ SortedMap, mutable } +import scala.collection.SortedMap import scala.language.{ implicitConversions, postfixOps } import scala.util.Try @@ -42,10 +40,6 @@ class PlaceholderMapper(metadataTermsFile: File)(implicit parameters: BaseParame type Table = ju.Collection[ju.Map[String, String]] - val metadataNames: ju.Properties = loadProperties(metadataTermsFile) - .doIfFailure { case e => logger.error(s"could not read the metadata terms in $metadataTermsFile", e) } - .getOrElse(new ju.Properties()) - def datasetToPlaceholderMap(dataset: Dataset): Try[PlaceholderMap] = { logger.debug("create placeholder map") @@ -55,17 +49,12 @@ class PlaceholderMapper(metadataTermsFile: File)(implicit parameters: BaseParame headerMap <- if (parameters.isSample) sampleHeader(emd) else header(emd) dansLogo = DansLogo -> encodeImage(dansLogoFile) - footer = FooterText -> footerText(footerTextFile) + drivenByData = DrivenByData -> encodeImage(drivenByDataFile) depositorMap = depositor(dataset.easyUser) - accessRightMap <- datasetAccessCategory(emd) + openAccess = OpenAccess -> boolean2Boolean(isOpenAccess(emd)) + termsLicenseMap <- termsLicenseMap(emd) embargoMap = embargo(emd) - dateTime = CurrentDateAndTime -> currentDateAndTime - metadata = MetadataTable -> metadataTable(emd, dataset.audiences, dataset.datasetID) - files @ (_, table) = FileTable -> filesTable(dataset.fileItems) - hasFiles = HasFiles -> boolean2Boolean(!table.isEmpty) - limitFiles = LimitFiles -> parameters.fileLimit.toString - shouldLimitFiles = ShouldLimitFiles -> boolean2Boolean(dataset.filesLimited) - } yield headerMap + dansLogo + footer ++ depositorMap ++ accessRightMap ++ embargoMap + dateTime + metadata + files + hasFiles + limitFiles + shouldLimitFiles + } yield headerMap + dansLogo + drivenByData ++ depositorMap + openAccess ++ termsLicenseMap ++ embargoMap } def header(emd: EasyMetadata): Try[PlaceholderMap] = Try { @@ -112,23 +101,11 @@ class PlaceholderMapper(metadataTermsFile: File)(implicit parameters: BaseParame ) } - def datasetAccessCategory(emd: EasyMetadata): Try[PlaceholderMap] = Try { + def isOpenAccess(emd: EasyMetadata): Boolean = { // access category in EMD may be null, in which case OPEN_ACCESS is the default value - val ac = Option(emd.getEmdRights.getAccessCategory).getOrElse(OPEN_ACCESS) - - val result = Map[KeywordMapping, List[AccessCategory]]( - OpenAccess -> List(OPEN_ACCESS, ANONYMOUS_ACCESS, FREELY_AVAILABLE), - OpenAccessForRegisteredUsers -> List(OPEN_ACCESS_FOR_REGISTERED_USERS), - OtherAccess -> List(ACCESS_ELSEWHERE, NO_ACCESS), - RestrictGroup -> List(GROUP_ACCESS), - RestrictRequest -> List(REQUEST_PERMISSION) - // because Velocity requires Java objects, we transform Scala's Boolean into a Java Boolean - ).mapValues(lst => boolean2Boolean(lst.contains(ac))) - - if (result.exists { case (_, bool) => bool == true }) - result - else - throw new IllegalArgumentException(s"The specified access category ($ac) does not map to any of these keywords.") + val accessCategory = Option(emd.getEmdRights.getAccessCategory).getOrElse(OPEN_ACCESS) + List(OPEN_ACCESS, ANONYMOUS_ACCESS, FREELY_AVAILABLE) + .contains(accessCategory) } def embargo(emd: EasyMetadata): PlaceholderMap = { @@ -140,82 +117,23 @@ class PlaceholderMapper(metadataTermsFile: File)(implicit parameters: BaseParame ) } - def currentDateAndTime: String = new DateTime().toString("YYYY-MM-dd HH:mm:ss") - - private val newLine = "
    " - - def metadataTable(emd: EasyMetadata, audiences: Seq[AudienceTitle], datasetID: => DatasetID): Table = { - def format(term: Term, items: mutable.Buffer[MetadataItem]): String = { - term.getName match { - case Term.Name.AUDIENCE => formatAudience(audiences, datasetID) - case Term.Name.ACCESSRIGHTS => formatDatasetAccessRights(items.head) - case Term.Name.SPATIAL => formatSpatials(items) - case Term.Name.LICENSE => getSpecifiedLicense(items) - .getOrElse(toLicense(emd.getEmdRights.getAccessCategory)) - case Term.Name.RELATION => formatRelations(items) - case _ => items.mkString(newLine) - } - } - - def getDisplayName(term: Term): String = { - val qualifiedName = term.getQualifiedName - Option(metadataNames.getProperty(qualifiedName)) - .getOrElse { - logger.warn(s"Could not find display name for a term with qualified name '$qualifiedName'") - qualifiedName - } - } - - emd.getTerms - .asScala - .map(term => (term, emd.getTerm(term).asScala)) - .filter { case (_, items) => items.nonEmpty } - .groupBy { case (term, _) => getDisplayName(term) } - .map { case (displayName, termsAndItems) => - // by definition of `groupBy`, `termsAndItems` cannot be empty - val termName = termsAndItems.head._1.getName - val value = termsAndItems - .map((format _).tupled) - .reduce(_ + newLine * 2 + _) - - // keep the Term.Name around for sorting according to the Enum order - termName -> Map( - MetadataKey -> displayName, - MetadataValue -> value - ).keywordMapAsJava - } - .sortedJavaCollection - } - - private def getSpecifiedLicense(licenseItems: mutable.Buffer[MetadataItem]): Option[String] = { - licenseItems.filterNot { - case s: BasicString if s.getValue == "accept" => true - case _ => false - }.map(_.toString).headOption - } - - private def toLicense(category: AccessCategory): String = { - category match { - case AccessCategory.OPEN_ACCESS => "http://creativecommons.org/publicdomain/zero/1.0/legalcode" - case _ => "http://dans.knaw.nl/en/about/organisation-and-policy/legal-information/DANSGeneralconditionsofuseUKDEF.pdf" - } - } - - private def formatRelations(items: mutable.Buffer[MetadataItem]): String = { - items.map { - case r: Relation => formatRelation(r) - case s => s.toString - }.mkString(newLine) + def termsLicenseMap(emd: EasyMetadata): Try[PlaceholderMap] = Try { + val url = emd.getEmdRights.getTermsLicense.asScala + .map(_.getValue) + .find(_.startsWith("http")) + .getOrElse(throw new IllegalArgumentException("Did not find a http...")) + val file = parameters.licenseLegalResource(url) + val extensionRegExp = ".[^.]+$" + val txtFile = file.toString.replaceAll(extensionRegExp, ".txt") + val baseFileName = new File(file).getName.replaceAll(extensionRegExp, "") + Map( + TermsLicenseUrl -> url, + TermsLicense -> baseFileName, + Appendix3 -> txtFile + ) } - def formatAudience(audiences: Seq[AudienceTitle], datasetID: => DatasetID): String = { - // may throw an UnsupportedOperationException - Try(audiences.reduce(_ + "; " + _)) - .doIfFailure { - case _: UnsupportedOperationException => logger.warn(s"Found a dataset with no audience: $datasetID. Returning an empty String instead.") - } - .getOrElse("") - } + def currentDateAndTime: String = new DateTime().toString("YYYY-MM-dd HH:mm:ss") def formatDatasetAccessRights(item: MetadataItem): String = { Try(AccessCategory.valueOf(item.toString)) // may throw an IllegalArgumentException @@ -235,60 +153,6 @@ class PlaceholderMapper(metadataTermsFile: File)(implicit parameters: BaseParame .getOrElse(item.toString) } - private def formatSpatials(items: mutable.Buffer[MetadataItem]): String = { - val basic = items.collect { - case s: BasicString => s.getValue - } - val spatial = items.collect { - case s: Spatial => formatEasSpatial(s).replace("\n", newLine) - } - (basic ++ spatial).mkString(newLine * 2) - } - - def formatEasSpatial(spatial: Spatial): String = { - val place = Option(spatial.getPlace).flatMap(_.getValue.toOption) - val point = Option(spatial.getPoint).map(formatPoint) - val box = Option(spatial.getBox).map(formatBox) - val polygonText = Option(spatial.getPolygons).flatMap(_.asScala.headOption).map(formatPolygon) - - (place.toList ::: point.toList ::: box.toList ::: polygonText.toList).mkString("\n") - } - - private def formatPoint(point: Point): String = { - val scheme = Option(point.getScheme).map("scheme = " + _ + ", ").getOrElse("") - val x = s"x = ${ point.getX }" - val y = s"y = ${ point.getY }" - - // note: no space between $scheme and $x: - // if $scheme is defined, it will do the space after it by itself; - // if $scheme is not defined, it doesn't require the extra space. - s"Point: $scheme$x, $y" - } - - private def formatBox(box: Box): String = { - val scheme = Option(box.getScheme).map("scheme = " + _ + ", ").getOrElse("") - val north = s"north = ${ box.getNorth }" - val east = s"east = ${ box.getEast }" - val south = s"south = ${ box.getSouth }" - val west = s"west = ${ box.getWest }" - - s"Box: $scheme$north, $east, $south, $west" - } - - private def formatPolygon(polygon: Polygon): String = { - s"""Polygon: - |To keep this agreement at a reasonable size the polygon coordinates are omitted. For a full listing of the polygons please contact DANS at info@dans.knaw.nl.""".stripMargin - } - - private def formatRelation(relation: Relation): String = { - val title = Option(relation.getSubjectTitle).flatMap(_.getValue.toOption.map("title = " +)) - val url = Option(relation.getSubjectLink).map("url = " +) - - title.map(t => url.fold(t)(u => s"$t, $u")) - .orElse(url) - .getOrElse("") - } - def formatFileAccessRights(accessRight: FileAccessRight.Value): String = { accessRight match { // @formatter:off @@ -301,20 +165,6 @@ class PlaceholderMapper(metadataTermsFile: File)(implicit parameters: BaseParame } } - def filesTable(fileItems: Seq[FileItem]): Table = { - fileItems - .map { case FileItem(path, accessibleTo, checkSum) => - val map = Map( - FilePath -> path, - FileChecksum -> checkSum.filterNot(_.isBlank).filterNot("none" ==).getOrElse(checkSumNotCalculated), - FileAccessibleTo -> formatFileAccessRights(accessibleTo) - ) - - (path, map.keywordMapAsJava) - } - .sortedJavaCollection - } - implicit class KeywordMapToJavaMap[Keyword <: KeywordMapping](map: Map[Keyword, String]) { def keywordMapAsJava: ju.Map[String, String] = map.map { case (k, v) => (k.keyword, v) }.asJava } diff --git a/src/main/scala/nl/knaw/dans/easy/agreement/internal/package.scala b/src/main/scala/nl/knaw/dans/easy/agreement/internal/package.scala index 4845774..a969167 100644 --- a/src/main/scala/nl/knaw/dans/easy/agreement/internal/package.scala +++ b/src/main/scala/nl/knaw/dans/easy/agreement/internal/package.scala @@ -25,20 +25,13 @@ import rx.lang.scala.Notification.{ OnCompleted, OnError, OnNext } import rx.lang.scala.{ Notification, Observable } import scala.language.postfixOps -import scala.util.Try import scala.xml.{ Elem, XML } package object internal { - type AudienceID = String - type AudienceTitle = String - - type FileID = String - type PlaceholderMap = Map[KeywordMapping, Object] val encoding: Charset = Charsets.UTF_8 - val checkSumNotCalculated = "------not-calculated------" def velocityProperties(implicit parameters: BaseParameters): Properties = { val p = new Properties @@ -58,8 +51,8 @@ package object internal { new File(parameters.templateResourceDir, "/dans_logo.png") } - def footerTextFile(implicit parameters: BaseParameters): File = { - new File(parameters.templateResourceDir, "/agreement_version.txt") + def drivenByDataFile(implicit parameters: BaseParameters): File = { + new File(parameters.templateResourceDir, "/DrivenByData.png") } def metadataTermsProperties(implicit parameters: BaseParameters): File = { @@ -85,55 +78,36 @@ package object internal { implicit class FileExtensions(val file: File) extends AnyVal { /** - * Copies a whole directory to a new location preserving the file dates. - *

    - * This method copies the specified directory and all its child - * directories and files to the specified destination. - * The destination is the new location and name of the directory. - *

    - * The destination directory is created if it does not exist. - * If the destination directory did exist, then this method merges - * the source with the destination, with the source taking precedence. - *

    - * Note: This method tries to preserve the files' last - * modified date/times using ``File#setLastModified(long)``, however - * it is not guaranteed that those operations will succeed. - * If the modification operation fails, no indication is provided. - * - * @param destDir the new directory, must not be ``null`` - */ + * Copies a whole directory to a new location preserving the file dates. + *

    + * This method copies the specified directory and all its child + * directories and files to the specified destination. + * The destination is the new location and name of the directory. + *

    + * The destination directory is created if it does not exist. + * If the destination directory did exist, then this method merges + * the source with the destination, with the source taking precedence. + *

    + * Note: This method tries to preserve the files' last + * modified date/times using ``File#setLastModified(long)``, however + * it is not guaranteed that those operations will succeed. + * If the modification operation fails, no indication is provided. + * + * @param destDir the new directory, must not be ``null`` + */ def copyDir(destDir: File): Unit = FileUtils.copyDirectory(file, destDir) /** - * Determines whether the ``parent`` directory contains the ``child`` element (a file or directory). - *

    - * Files are normalized before comparison. - *

    - * - * Edge cases: - *
      - *
    • A ``directory`` must not be null: if null, throw IllegalArgumentException
      • - *
    • A ``directory`` must be a directory: if not a directory, throw IllegalArgumentException
      • - *
    • A directory does not contain itself: return false
      • - *
    • A null child file is not contained in any parent: return false
      • - *
    - * - * @param child the file to consider as the child. - * @return true is the candidate leaf is under by the specified composite. False otherwise. - */ - def directoryContains(child: File): Boolean = FileUtils.directoryContains(file, child) - - /** - * Deletes a directory recursively. - */ + * Deletes a directory recursively. + */ def deleteDirectory(): Unit = FileUtils.deleteDirectory(file) /** - * Reads the contents of a file into a String using the default encoding for the VM. - * The file is always closed. - * - * @return the file contents, never ``null`` - */ + * Reads the contents of a file into a String using the default encoding for the VM. + * The file is always closed. + * + * @return the file contents, never ``null`` + */ def read(encoding: Charset = Charsets.UTF_8): String = FileUtils.readFileToString(file, encoding) } @@ -141,7 +115,7 @@ package object internal { def loadXML: Elem = XML.load(stream) } - // TODO not used here anymore, but useful for debugging purposed. Maybe we can migrate this to a EASY-Utils project? + // TODO not used here anymore, but useful for debugging purposes. implicit class ObservableDebug[T](val observable: Observable[T]) extends AnyVal { def debugThreadName(s: String = "")(implicit logger: Logger): Observable[T] = { @@ -154,7 +128,7 @@ package object internal { } observable.materialize - .doOnEach(o => logger.debug(s"$s: ${notificationKind(o)} - ${Thread.currentThread().getName}")) + .doOnEach(o => logger.debug(s"$s: ${ notificationKind(o) } - ${ Thread.currentThread().getName }")) .dematerialize } @@ -164,14 +138,4 @@ package object internal { .dematerialize } } - - def loadProperties(file: File): Try[Properties] = { - resource.Using.fileInputStream(file) - .map(fis => { - val props = new Properties - props.load(fis) - props - }) - .tried - } } diff --git a/src/test/resources/datasetloader/emd.xml b/src/test/resources/datasetloader/emd.xml index 688ee6d..2cd4429 100644 --- a/src/test/resources/datasetloader/emd.xml +++ b/src/test/resources/datasetloader/emd.xml @@ -2,8 +2,4 @@ descr foo bar - - aud1 - aud2 - diff --git a/src/test/resources/debug-config/application.properties b/src/test/resources/debug-config/application.properties index 5f139f8..ec87823 100644 --- a/src/test/resources/debug-config/application.properties +++ b/src/test/resources/debug-config/application.properties @@ -1,12 +1,8 @@ fcrepo.url=http://deasy.dans.knaw.nl:8080/fedora fcrepo.user=fedoraAdmin fcrepo.password=fedoraAdmin -fsrdb.db-connection-url=jdbc:postgresql://deasy.dans.knaw.nl:5432/easy_db -fsrdb.db-connection-username=easy_webui -fsrdb.db-connection-password=easy_webui auth.ldap.url=ldap://deasy.dans.knaw.nl auth.ldap.user=cn=ldapadmin,dc=dans,dc=knaw,dc=nl auth.ldap.password=ldapadmin agreement.resources=home/res -agreement.fileLimit=3 daemon.http.port=20130 diff --git a/src/test/scala/nl/knaw/dans/easy/agreement/AgreementCreatorServletSpec.scala b/src/test/scala/nl/knaw/dans/easy/agreement/AgreementCreatorServletSpec.scala index 1578c2e..b0fb0b7 100644 --- a/src/test/scala/nl/knaw/dans/easy/agreement/AgreementCreatorServletSpec.scala +++ b/src/test/scala/nl/knaw/dans/easy/agreement/AgreementCreatorServletSpec.scala @@ -47,14 +47,10 @@ class AgreementCreatorServletSpec extends FlatSpec addProperty("fcrepo.url", "http://localhost:8080/fedora") addProperty("fcrepo.user", "-") addProperty("fcrepo.password", "-") - addProperty("fsrdb.db-connection-url", "") - addProperty("fsrdb.db-connection-username", "-") - addProperty("fsrdb.db-connection-password", "-") addProperty("auth.ldap.url", "ldap://localhost") addProperty("auth.ldap.user", "") addProperty("auth.ldap.password", "") addProperty("agreement.resources", "") - addProperty("agreement.fileLimit", "3") }) } } diff --git a/src/test/scala/nl/knaw/dans/easy/agreement/TemplateResolverSpec.scala b/src/test/scala/nl/knaw/dans/easy/agreement/TemplateResolverSpec.scala new file mode 100644 index 0000000..ed796a4 --- /dev/null +++ b/src/test/scala/nl/knaw/dans/easy/agreement/TemplateResolverSpec.scala @@ -0,0 +1,164 @@ +/** + * Copyright (C) 2016 DANS - Data Archiving and Networked Services (info@dans.knaw.nl) + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package nl.knaw.dans.easy.agreement + +import java.io.{ ByteArrayOutputStream, File, FileOutputStream, OutputStream } +import java.util.Arrays.asList + +import nl.knaw.dans.common.lang.dataset.AccessCategory +import nl.knaw.dans.easy.agreement.internal._ +import nl.knaw.dans.pf.language.emd.Term.Name +import nl.knaw.dans.pf.language.emd._ +import nl.knaw.dans.pf.language.emd.types.{ BasicString, IsoDate } +import org.joda.time.DateTime +import org.scalamock.scalatest.MockFactory +import org.scalatest.BeforeAndAfterAll +import org.scalatest.prop.TableDrivenPropertyChecks + +import scala.util.{ Failure, Success } + +class TemplateResolverSpec extends UnitSpec with MockFactory with TableDrivenPropertyChecks with BeforeAndAfterAll { + trait MockEasyMetadata extends EasyMetadata { + def toString(x: String, y: Name): String = "" + + def toString(x: String, y: Term): String = "" + + def toString(x: String, y: MDContainer): String = "" + + def toString(x: String): String = "" + } + + private val templateResourceDir = new java.io.File(testDir, "res") + private val properties = new File(templateResourceDir, "MetadataTestTerms.properties") + private val datasetId = "easy:12" + private val user = EasyUser( + name = "N.O. Body", + organization = "Eidgenössische Technische Hochschule", + address = "Rämistrasse 101", + postalCode = "8092", + city = " Zürich", + country = "Schweiz", + telephone = "+41 44 632 11 11", + email = "nobody@dans.knaw.nl" + ) + + override def beforeAll() { + testDir.mkdirs() + super.beforeAll() + testDir.deleteDirectory() + testDir.mkdirs() + new File("src/main/assembly/dist/res").copyDir(new File(testDir, "res")) + new File("target/easy-licenses/licenses").copyDir(new File(testDir, "res/template/licenses")) + } + + "createTemplate" should "find all place holders" in { + forEvery(for { + rights <- Seq(AccessCategory.OPEN_ACCESS, AccessCategory.REQUEST_PERMISSION) + isSample <- Seq(true, false) + available <- Seq(new DateTime, (new DateTime).plusYears(1)).map(new IsoDate(_)) + } yield (isSample, rights, available)) { + case (isSample, rights, available) => + create( + isSample, + mockDataset(rights, isSample, available), + new FileOutputStream(docName(isSample, rights, available)) + ) shouldBe Success(()) + } + } + + it should "not find a license url" in { + val dataset = mockDataset(AccessCategory.OPEN_ACCESS, isSample = false, new IsoDate(), new BasicString("accept")) + val outputStream = new ByteArrayOutputStream() + + create(isSample = false, dataset, outputStream) should matchPattern { + case Failure(e: IllegalArgumentException) if (e.getMessage == "Did not find a http...") => + } + } + + it should "not find the resource for Appendix 3" in { + val dataset = mockDataset(AccessCategory.OPEN_ACCESS, isSample = false, new IsoDate(), new BasicString("http://dans.knaw.nl")) + val outputStream = new ByteArrayOutputStream() + + create(isSample = false, dataset, outputStream) should matchPattern { + case Failure(e: java.lang.IllegalArgumentException) if (e.getMessage == "No legal text found for http://dans.knaw.nl") => + } + } + + it should "properly format all types of licenses" in { + forEvery(Seq( + new BasicString("http://creativecommons.org/licenses/by-nc-sa/3.0") + -> """BY-NC-SA-3.0 : http://creativecommons.org/licenses/by-nc-sa/3.0""", + new BasicString("https://creativecommons.org/licenses/by-nc-sa/3.0") + -> """BY-NC-SA-3.0 : https://creativecommons.org/licenses/by-nc-sa/3.0""", + new BasicString("https://www.creativecommons.org/licenses/by-nc-sa/3.0") + -> """BY-NC-SA-3.0 : https://www.creativecommons.org/licenses/by-nc-sa/3.0""", + new BasicString("http://www.creativecommons.org/licenses/by-nc-sa/3.0") + -> """BY-NC-SA-3.0 : http://www.creativecommons.org/licenses/by-nc-sa/3.0""", + )) { + case (emdRightsTermsLicense, expected) => + val dataset = mockDataset(AccessCategory.OPEN_ACCESS, isSample = false, new IsoDate(), emdRightsTermsLicense) + val outputStream = new ByteArrayOutputStream() + + create(isSample = false, dataset, outputStream) shouldBe Success(()) + + val lines = outputStream.toString.split("\n") + // .filter(_.contains("""class="choice">""")) // TODO less output in case of failure, but it's no longer on a single line + // lines should have length (1) // detected a forgotten #else in the template + lines.mkString("\n", "\n", "\n") should include(expected) + } + } + + private def create(isSample: Boolean, dataset: Dataset, outputStream: OutputStream) = { + implicit val parameters: BaseParameters = new BaseParameters(templateResourceDir, datasetId, isSample) + new PlaceholderMapper(properties) + .datasetToPlaceholderMap(dataset) + .flatMap(placeholders => + new VelocityTemplateResolver(velocityProperties) + .createTemplate(outputStream, placeholders) + ) + } + + private def docName(isSample: Boolean, rights: AccessCategory, available: IsoDate): String = { + val part1 = rights match { + case AccessCategory.OPEN_ACCESS => "OA" + case _ => "RA" // restricted as in appendix1.html + } + val part2 = if (isSample) "sample-" + else "" + s"$testDir/$part1-$part2$available.html" + } + + private def mockDataset(openaccess: AccessCategory, isSample: Boolean, dateAvailable: IsoDate, + emdRightsTermsLicense: BasicString = new BasicString("http://creativecommons.org/licenses/by-nc-sa/3.0") + ): Dataset = { + val emd = mock[MockEasyMetadata] + val date = mock[EmdDate] + val rights = mock[EmdRights] + emd.getPreferredTitle _ expects() returning "about testing" + emd.getEmdRights _ expects() returning rights twice() + emd.getEmdDate _ expects() returning date anyNumberOfTimes() + rights.getAccessCategory _ expects() returning openaccess + rights.getTermsLicense _ expects() returning asList(emdRightsTermsLicense) + date.getEasDateSubmitted _ expects() returning asList(new IsoDate("1992-07-30")) + date.getEasAvailable _ expects() returning asList(dateAvailable) anyNumberOfTimes() + if (!isSample) { + val emdIdentifier = mock[EmdIdentifier] + emd.getEmdIdentifier _ expects() returning emdIdentifier anyNumberOfTimes() + emdIdentifier.getDansManagedDoi _ expects() returning "10.17026/test-dans-2xg-umq8" anyNumberOfTimes() + } + Dataset(datasetID = "easy:12", emd, user).validate + } +} diff --git a/src/test/scala/nl/knaw/dans/easy/agreement/internal/DatasetLoaderSpec.scala b/src/test/scala/nl/knaw/dans/easy/agreement/internal/DatasetLoaderSpec.scala index bee3186..ed8d8a3 100644 --- a/src/test/scala/nl/knaw/dans/easy/agreement/internal/DatasetLoaderSpec.scala +++ b/src/test/scala/nl/knaw/dans/easy/agreement/internal/DatasetLoaderSpec.scala @@ -16,12 +16,9 @@ package nl.knaw.dans.easy.agreement.internal import java.io.File -import java.sql.{ Connection, PreparedStatement, ResultSet } -import java.util -import javax.naming.directory.BasicAttributes +import javax.naming.directory.BasicAttributes import nl.knaw.dans.easy.agreement._ -import nl.knaw.dans.pf.language.emd._ import org.apache.commons.io.{ FileUtils, IOUtils } import org.scalamock.scalatest.MockFactory import org.scalatest.{ BeforeAndAfter, BeforeAndAfterAll } @@ -35,7 +32,6 @@ class DatasetLoaderSpec extends UnitSpec with MockFactory with BeforeAndAfter wi private val fedoraMock = mock[Fedora] private val ldapMock = mock[Ldap] - private val fsrdbMock = mock[Connection] val (userAttributes, expectedUser) = { val attrs = new BasicAttributes @@ -56,8 +52,6 @@ class DatasetLoaderSpec extends UnitSpec with MockFactory with BeforeAndAfter wi implicit val parameters: DatabaseParameters = new DatabaseParameters { override val fedora: Fedora = fedoraMock override val ldap: Ldap = ldapMock - override val fsrdb: Connection = fsrdbMock - override val fileLimit: Int = 2 } before { @@ -122,11 +116,6 @@ class DatasetLoaderSpec extends UnitSpec with MockFactory with BeforeAndAfter wi val id = "testID" val depID = "depID" val user = EasyUser("name", "org", "addr", "pc", "city", "cntr", "phone", "mail") - val audiences = Seq("aud1", "aud2") - val files = Seq( - FileItem("path1", FileAccessRight.RESTRICTED_GROUP, Some("chs1")), - FileItem("path2", FileAccessRight.KNOWN, Some("chs2")) - ) val amdStream = IOUtils.toInputStream({depID}.toString) val emdStream = FileUtils.openInputStream(new File(testDir, "datasetloader/emd.xml")) @@ -134,151 +123,23 @@ class DatasetLoaderSpec extends UnitSpec with MockFactory with BeforeAndAfter wi fedoraMock.getAMD _ expects id returning Observable.just(amdStream) fedoraMock.getEMD _ expects id returning Observable.just(emdStream) - val loader = new DatasetLoaderImpl { + val loader: DatasetLoaderImpl = new DatasetLoaderImpl { override def getUserById(depositorID: DepositorID): Observable[EasyUser] = { if (depositorID == depID) Observable.just(user) else fail(s"not the correct depositorID, was $depositorID, should be $depID") } - - override def getAudiences(a: EmdAudience): Observable[AudienceTitle] = { - if (a.getDisciplines.asScala.map(_.getValue) == audiences) Observable.from(audiences) - else fail("not the correct audiences") - } - - override def getFilesInDataset(did: DatasetID): Observable[FileItem] = { - if (did == id) Observable.from(files) - else fail(s"not the correct datasetID, was $did, should be $id") - } - - override def countFiles(datasetID: DatasetID): Observable[Int] = { - Observable.just(3) - } } - val testObserver = TestSubscriber[(DatasetID, Seq[String], EasyUser, Seq[AudienceTitle], Seq[FileItem], Boolean)]() + val testObserver = TestSubscriber[(DatasetID, Seq[String], EasyUser)]() loader.getDatasetById(id) // there is no equals defined for the emd, so I need to unpack here - .map { case Dataset(datasetID, emd, usr, titles, fileItems, allFilesListed) => - (datasetID, emd.getEmdDescription.getDcDescription.asScala.map(_.getValue), usr, titles, fileItems, allFilesListed) } + .map { case Dataset(datasetID, emd, usr) => + (datasetID, emd.getEmdDescription.getDcDescription.asScala.map(_.getValue), usr) + } .subscribe(testObserver) testObserver.awaitTerminalEvent() - testObserver.assertValue((id, Seq("descr foo bar"), user, audiences, files, true)) - testObserver.assertNoErrors() - testObserver.assertCompleted() - } - - "getFilesInDataset" should "return the files contained in the dataset corresponding to the given datasetID" in { - val id = "testID" - val pid1 = "pid1" - val pid2 = "pid2" - val fi1 @ FileItem(path1, accTo1, Some(chcksm1)) = FileItem("path1", FileAccessRight.NONE, Some("chcksm1")) - val fi2 @ FileItem(path2, accTo2, _) = FileItem("path2", FileAccessRight.KNOWN, None) - - val mockedPrepStatement = mock[PreparedStatement] - val mockedResultSet = mock[ResultSet] - - inSequence { - (fsrdbMock.prepareStatement(_: String)) expects * returning mockedPrepStatement - mockedPrepStatement.setString _ expects(1, id) - mockedPrepStatement.setInt _ expects(2, 2) - mockedPrepStatement.executeQuery _ expects() returning mockedResultSet - mockedResultSet.next _ expects() returning true - (mockedResultSet.getString(_: String)) expects "pid" returning pid1 - (mockedResultSet.getString(_: String)) expects "path" returning path1 - (mockedResultSet.getString(_: String)) expects "sha1checksum" returning chcksm1 - (mockedResultSet.getString(_: String)) expects "accessible_to" returning accTo1.toString - mockedResultSet.next _ expects() returning true - (mockedResultSet.getString(_: String)) expects "pid" returning pid2 - (mockedResultSet.getString(_: String)) expects "path" returning path2 - (mockedResultSet.getString(_: String)) expects "sha1checksum" returning "null" - (mockedResultSet.getString(_: String)) expects "accessible_to" returning accTo2.toString - mockedResultSet.next _ expects() returning false - mockedResultSet.close _ expects() - mockedPrepStatement.close _ expects() - } - - val loader = DatasetLoaderImpl() - val testObserver = TestSubscriber[Set[FileItem]]() - loader.getFilesInDataset(id).toSet.subscribe(testObserver) - - testObserver.awaitTerminalEvent() - testObserver.assertValue(Set(fi1, fi2)) - testObserver.assertNoErrors() - testObserver.assertCompleted() - } - - "countFiles" should "return the number of files contained in the dataset corresponding to the given datasetID" in { - val id = "testID" - val fileCount = 500 - - val mockedPrepStatement = mock[PreparedStatement] - val mockedResultSet = mock[ResultSet] - - inSequence { - (fsrdbMock.prepareStatement(_: String)) expects * returning mockedPrepStatement - mockedPrepStatement.setString _ expects(1, id) - mockedPrepStatement.executeQuery _ expects() returning mockedResultSet - mockedResultSet.next _ expects() returning true - (mockedResultSet.getInt(_: String)) expects "count" returning fileCount - mockedResultSet.close _ expects() - mockedPrepStatement.close _ expects() - } - - val loader = DatasetLoaderImpl() - val testObserver = TestSubscriber[Int]() - loader.countFiles(id).subscribe(testObserver) - - testObserver.awaitTerminalEvent() - testObserver.assertValue(fileCount) - testObserver.assertNoErrors() - testObserver.assertCompleted() - } - - "getAudience" should "search the title for each audienceID in the EmdAudience in Fedora" in { - val is = IOUtils.toInputStream(title1.toString) - val (id1, title1) = ("id1", "title1") - - fedoraMock.getDC _ expects id1 returning Observable.just(is) - - val loader = DatasetLoaderImpl() - val testObserver = TestSubscriber[String]() - loader.getAudience(id1).subscribe(testObserver) - - testObserver.awaitTerminalEvent() - testObserver.assertValues(title1) - testObserver.assertNoErrors() - testObserver.assertCompleted() - } - - "getAudiences" should "search the title for each audienceID in the EmdAudience in Fedora" in { - val (id1, title1) = ("id1", "title1") - val (id2, title2) = ("id2", "title2") - val (id3, title3) = ("id3", "title3") - val audience = mock[EmdAudience] - - audience.getValues _ expects() returning util.Arrays.asList(id1, id2, id3) - - // can't do mocking due to concurrency issues - val loader = new DatasetLoaderImpl { - - var counter = 0 - - override def getAudience(audienceID: AudienceID): Observable[AudienceID] = { - counter += 1 - counter match { - case 1 => Observable.just(title1) - case 2 => Observable.just(title2) - case 3 => Observable.just(title3) - case _ => throw new IllegalStateException(s"Called this method too many times. audienceID = $audienceID") - } - } - } - val testObserver = TestSubscriber[String]() - loader.getAudiences(audience).subscribe(testObserver) - - testObserver.assertValues(title1, title2, title3) + testObserver.assertValue((id, Seq("descr foo bar"), user)) testObserver.assertNoErrors() testObserver.assertCompleted() - loader.counter shouldBe 3 } } diff --git a/src/test/scala/nl/knaw/dans/easy/agreement/internal/PlaceholderMapperSpec.scala b/src/test/scala/nl/knaw/dans/easy/agreement/internal/PlaceholderMapperSpec.scala index dab0f30..2ddc46e 100644 --- a/src/test/scala/nl/knaw/dans/easy/agreement/internal/PlaceholderMapperSpec.scala +++ b/src/test/scala/nl/knaw/dans/easy/agreement/internal/PlaceholderMapperSpec.scala @@ -16,14 +16,12 @@ package nl.knaw.dans.easy.agreement.internal import java.io.File -import java.net.URI import java.{ util => ju } import nl.knaw.dans.common.lang.dataset.AccessCategory -import nl.knaw.dans.easy.agreement.{ FileAccessRight, FileItem, UnitSpec } +import nl.knaw.dans.easy.agreement.{ FileAccessRight, UnitSpec } import nl.knaw.dans.pf.language.emd.Term.{ Name, Namespace } import nl.knaw.dans.pf.language.emd._ -import nl.knaw.dans.pf.language.emd.types.Spatial.{ Box, Point } import nl.knaw.dans.pf.language.emd.types._ import org.joda.time.DateTime import org.scalamock.scalatest.MockFactory @@ -55,8 +53,7 @@ class PlaceholderMapperSpec extends UnitSpec with MockFactory with BeforeAndAfte isSample = false, fedora = null, ldap = null, - fsrdb = null, - fileLimit = 3) + ) before { new File(getClass.getResource("/placeholdermapper/").toURI).copyDir(parameters.templateResourceDir) @@ -129,8 +126,7 @@ class PlaceholderMapperSpec extends UnitSpec with MockFactory with BeforeAndAfte isSample = true, fedora = null, ldap = null, - fsrdb = null, - fileLimit = 3) + ) val dates = ju.Arrays.asList(new IsoDate("1992-07-30"), new IsoDate("2016-07-30")) emd.getEmdIdentifier _ expects() never() @@ -156,8 +152,7 @@ class PlaceholderMapperSpec extends UnitSpec with MockFactory with BeforeAndAfte isSample = true, fedora = null, ldap = null, - fsrdb = null, - fileLimit = 3) + ) emd.getEmdIdentifier _ expects() never() ident.getDansManagedDoi _ expects() never() @@ -220,119 +215,47 @@ class PlaceholderMapperSpec extends UnitSpec with MockFactory with BeforeAndAfte "accessRights" should "map an Open Access category to an OpenAccess keyword" in { expectEmdRights(AccessCategory.OPEN_ACCESS) - - inside(testInstance.datasetAccessCategory(emd)) { - case Success(map) => map should contain theSameElementsAs List( - (OpenAccess, true), - (OpenAccessForRegisteredUsers, false), - (OtherAccess, false), - (RestrictGroup, false), - (RestrictRequest, false)) - } + testInstance.isOpenAccess(emd) shouldBe true } it should "map an Anonymous Access category to an OpenAccess keyword" in { expectEmdRights(AccessCategory.ANONYMOUS_ACCESS) - - inside(testInstance.datasetAccessCategory(emd)) { - case Success(map) => map should contain theSameElementsAs List( - (OpenAccess, true), - (OpenAccessForRegisteredUsers, false), - (OtherAccess, false), - (RestrictGroup, false), - (RestrictRequest, false)) - } + testInstance.isOpenAccess(emd) shouldBe true } it should "map a Freely Available category to an OpenAccess keyword" in { expectEmdRights(AccessCategory.FREELY_AVAILABLE) - - inside(testInstance.datasetAccessCategory(emd)) { - case Success(map) => map should contain theSameElementsAs List( - (OpenAccess, true), - (OpenAccessForRegisteredUsers, false), - (OtherAccess, false), - (RestrictGroup, false), - (RestrictRequest, false)) - } + testInstance.isOpenAccess(emd) shouldBe true } it should "map an Open Access For Registered Users category to an OpenAccessForRegisteredUsers keyword" in { expectEmdRights(AccessCategory.OPEN_ACCESS_FOR_REGISTERED_USERS) - - inside(testInstance.datasetAccessCategory(emd)) { - case Success(map) => map should contain theSameElementsAs List( - (OpenAccess, false), - (OpenAccessForRegisteredUsers, true), - (OtherAccess, false), - (RestrictGroup, false), - (RestrictRequest, false)) - } + testInstance.isOpenAccess(emd) shouldBe false } it should "map a Group Access category to an RestrictGroup keyword" in { expectEmdRights(AccessCategory.GROUP_ACCESS) - - inside(testInstance.datasetAccessCategory(emd)) { - case Success(map) => map should contain theSameElementsAs List( - (OpenAccess, false), - (OpenAccessForRegisteredUsers, false), - (OtherAccess, false), - (RestrictGroup, true), - (RestrictRequest, false)) - } + testInstance.isOpenAccess(emd) shouldBe false } it should "map a Request Permission category to an RestrictRequest keyword" in { expectEmdRights(AccessCategory.REQUEST_PERMISSION) - - inside(testInstance.datasetAccessCategory(emd)) { - case Success(map) => map should contain theSameElementsAs List( - (OpenAccess, false), - (OpenAccessForRegisteredUsers, false), - (OtherAccess, false), - (RestrictGroup, false), - (RestrictRequest, true)) - } + testInstance.isOpenAccess(emd) shouldBe false } it should "map an Access Elsewhere category to an OtherAccess keyword" in { expectEmdRights(AccessCategory.ACCESS_ELSEWHERE) - - inside(testInstance.datasetAccessCategory(emd)) { - case Success(map) => map should contain theSameElementsAs List( - (OpenAccess, false), - (OpenAccessForRegisteredUsers, false), - (OtherAccess, true), - (RestrictGroup, false), - (RestrictRequest, false)) - } + testInstance.isOpenAccess(emd) shouldBe false } it should "map a No Access category to an OtherAccess keyword" in { expectEmdRights(AccessCategory.NO_ACCESS) - - inside(testInstance.datasetAccessCategory(emd)) { - case Success(map) => map should contain theSameElementsAs List( - (OpenAccess, false), - (OpenAccessForRegisteredUsers, false), - (OtherAccess, true), - (RestrictGroup, false), - (RestrictRequest, false)) - } + testInstance.isOpenAccess(emd) shouldBe false } it should "map a null value to an OpenAccess keyword" in { - expectEmdRights(AccessCategory.OPEN_ACCESS) - - inside(testInstance.datasetAccessCategory(emd)) { - case Success(map) => map should contain theSameElementsAs List( - (OpenAccess, true), - (OpenAccessForRegisteredUsers, false), - (OtherAccess, false), - (RestrictGroup, false), - (RestrictRequest, false)) - } + expectEmdRights(accessCategory = null) + testInstance.isOpenAccess(emd) shouldBe true } "embargo" should "give the embargo keyword mappings with UnderEmbargo=true when there is an embargo" in { @@ -369,171 +292,6 @@ class PlaceholderMapperSpec extends UnitSpec with MockFactory with BeforeAndAfte ) } - "metadataTable" should "give a mapping for the metadata elements in the dataset" in { - val ccBy = "https://creativecommons.org/licenses/by/4.0/legalcode" - - val audienceTerm = new Term(Name.AUDIENCE, Namespace.DCTERMS) - val mediumTerm = new Term(Name.MEDIUM, Namespace.DC) - val abstractTerm = new Term(Name.ABSTRACT, Namespace.EAS) - val terms = Set(audienceTerm, accessRightsTerm, licenseTerm, mediumTerm, abstractTerm).asJava - - val anonymous = metadataItemMock("ANONYMOUS_ACCESS") - val open = metadataItemMock("OPEN_ACCESS") - val item1 = metadataItemMock("item1") - val item4 = metadataItemMock("item4") - val item5 = metadataItemMock("item5") - val item6 = metadataItemMock("item6") - - val audienceItems = List(item1, anonymous).asJava - val accessRightItems = List(anonymous, open).asJava - val mediumItems = List(item4, item5, item6).asJava - val abstractItems = ju.Collections.emptyList[MetadataItem]() - - emd.getTerms _ expects() returning terms - emd.getTerm _ expects audienceTerm returning audienceItems - emd.getTerm _ expects accessRightsTerm returning accessRightItems - emd.getTerm _ expects mediumTerm returning mediumItems - emd.getTerm _ expects abstractTerm returning abstractItems - - expectLicenses(Seq("accept", ccBy)) - - testInstance.metadataTable(emd, Seq("abc", "def"), "datasetID:1234").asScala.toList shouldBe List( - Map(MetadataKey.keyword -> "ghi", MetadataValue.keyword -> "item4
    item5
    item6").asJava, - Map(MetadataKey.keyword -> "Access rights", MetadataValue.keyword -> "Anonymous").asJava, - Map(MetadataKey.keyword -> "License", MetadataValue.keyword -> ccBy).asJava, - Map(MetadataKey.keyword -> "Audience", MetadataValue.keyword -> "abc; def").asJava, - ) - } - - it should "use the qualified names on invalid input" in { - // valid input would have DCTERMS for both - val audienceTerm = new Term(Name.AUDIENCE, Namespace.DC) - val accessRightsTerm = new Term(Name.ACCESSRIGHTS, Namespace.DC) - val items = List(metadataItemMock("ANONYMOUS_ACCESS")).asJava - emd.getTerms _ expects() returning Set(audienceTerm, accessRightsTerm).asJava - emd.getTerm _ expects audienceTerm returning items - emd.getTerm _ expects accessRightsTerm returning items - - testInstance.metadataTable(emd, Seq("abc", "def"), "datasetID:1234").asScala.toList shouldBe List( - Map(MetadataKey.keyword -> "DC.ACCESSRIGHTS", MetadataValue.keyword -> "Anonymous").asJava, - Map(MetadataKey.keyword -> "DC.AUDIENCE", MetadataValue.keyword -> "abc; def").asJava, - ) - } - - it should "give a mapping with a cc0 license" in { - emd.getTerms _ expects() returning Set(licenseTerm, accessRightsTerm).asJava - expectLicenses(Seq("accept")) - expectEmdRights(AccessCategory.OPEN_ACCESS) - expectRightsTerms(AccessCategory.OPEN_ACCESS) - - testInstance.metadataTable(emd, Seq("abc", "def"), "datasetID:1234").asScala should contain - Map( - MetadataKey.keyword -> "license", - MetadataValue.keyword -> "http://creativecommons.org/publicdomain/zero/1.0/legalcode", - ).asJava - } - - it should "give a mapping with a DANS license" in { - emd.getTerms _ expects() returning Set(licenseTerm, accessRightsTerm).asJava - expectLicenses(Seq("accept")) - expectEmdRights(AccessCategory.GROUP_ACCESS) - expectRightsTerms(AccessCategory.GROUP_ACCESS) - - testInstance.metadataTable(emd, Seq("abc", "def"), "datasetID:1234").asScala should contain - Map( - MetadataKey.keyword -> "license", - MetadataValue.keyword -> "http://dans.knaw.nl/en/about/organisation-and-policy/legal-information/DANSGeneralconditionsofuseUKDEF.pdf", - ).asJava - } - - it should "map an empty license list to a default license" in { - emd.getTerms _ expects() returning Set(licenseTerm, accessRightsTerm).asJava - val items: Seq[MetadataItem] = Seq[MetadataItem]() - emd.getTerm _ expects licenseTerm returning items.asJava - - expectRightsTerms(AccessCategory.REQUEST_PERMISSION) - - testInstance.metadataTable(emd, Seq("abc", "def"), "datasetID:1234").asScala should contain - Map( - MetadataKey.keyword -> "license", - MetadataValue.keyword -> "http://dans.knaw.nl/en/about/organisation-and-policy/legal-information/DANSGeneralconditionsofuseUKDEF.pdf", - ).asJava - } - - it should "map just access rights to a default license" in { - emd.getTerms _ expects() returning Set(accessRightsTerm).asJava - expectRightsTerms(AccessCategory.OPEN_ACCESS) - - testInstance.metadataTable(emd, Seq("abc", "def"), "datasetID:1234").asScala should contain - Map( - MetadataKey.keyword -> "License", - MetadataValue.keyword -> "http://dans.knaw.nl/en/about/organisation-and-policy/legal-information/DANSGeneralconditionsofuseUKDEF.pdf", - ).asJava - } - - it should "map the specified license" in { - val ccBy = "https://creativecommons.org/licenses/by/4.0/legalcode" - emd.getTerms _ expects() returning Set(licenseTerm).asJava - expectLicenses(Seq(ccBy)) - - testInstance.metadataTable(emd, Seq("abc", "def"), "datasetID:1234").asScala.toSeq shouldBe Seq( - Map( - MetadataKey.keyword -> "License", - MetadataValue.keyword -> ccBy, - ).asJava, - ) - } - - it should "map relations" in { - val term = new Term(Name.RELATION, Namespace.DC) - emd.getTerms _ expects() returning Set(term).asJava - emd.getTerm _ expects term returning Seq( - new BasicString("foo"), - new Relation("bar"), - new Relation("rabarbera", new URI("http://xx.dans.knaw.nl/yy")), - ).asJava - - testInstance.metadataTable(emd, Seq("abc", "def"), "datasetID:1234").asScala.toSeq shouldBe Seq( - Map( - MetadataKey.keyword -> "Relation", - MetadataValue.keyword -> Seq( - "foo", - "title = bar", - "title = rabarbera, url = http://xx.dans.knaw.nl/yy" - ).mkString("
    "), - ).asJava, - ) - } - - it should "map spatials" in { - val exterior = new PolygonPart("main triangle", ju.Arrays.asList( - new PolygonPoint("52.08110", "4.34521"), - new PolygonPoint("52.08071", "4.34422"), - new PolygonPoint("52.07913", "4.34332"), - new PolygonPoint("52.08110", "4.34521") - )) - val term = new Term(Name.SPATIAL, Namespace.DCTERMS) - emd.getTerms _ expects() returning Set(term).asJava - emd.getTerm _ expects term returning Seq( - new BasicString("foo"), - new Spatial("Amsterdam", new Point("RD", "1", "2")), - new Spatial("Amsterdam", new Box("RD", "463001", "155001", "462999", "154999")), - new Spatial("foo", new Polygon("bar", exterior, List().asJava)), - ).asJava - - testInstance.metadataTable(emd, Seq("abc", "def"), "datasetID:1234").asScala.toSeq shouldBe Seq( - Map( - MetadataKey.keyword -> "Spatial", - MetadataValue.keyword -> Seq( - "foo", - "Amsterdam
    Point: scheme = RD, x = 1, y = 2", - "Amsterdam
    Box: scheme = RD, north = 463001, east = 155001, south = 462999, west = 154999", - """foo
    Polygon:
    To keep this agreement at a reasonable size the polygon coordinates are omitted. For a full listing of the polygons please contact DANS at info@dans.knaw.nl.""", - ).mkString("

    "), - ).asJava, - ) - } - private val licenseTerm = new Term(Name.LICENSE, Namespace.DCTERMS) private def expectLicenses(values: Seq[String]) = { @@ -553,18 +311,6 @@ class PlaceholderMapperSpec extends UnitSpec with MockFactory with BeforeAndAfte emd.getTerm _ expects accessRightsTerm returning items.asJava } - "formatAudience" should "combine the audiences separated by " in { - testInstance.formatAudience(Seq("abc", "def"), "datasetID:1234") shouldBe "abc; def" - } - - it should "do the same with one element in the audience" in { - testInstance.formatAudience(Seq("abc"), "datasetID:1234") shouldBe "abc" - } - - it should "yield an Observable with one empty String when there is no audience" in { - testInstance.formatAudience(Seq.empty, "datasetID:1234") shouldBe "" - } - "formatAccessRights" should "return a String representation of the access category ANONYMOUS_ACCESS" in { testInstance.formatDatasetAccessRights(metadataItemMock("ANONYMOUS_ACCESS")) shouldBe "Anonymous" } @@ -620,18 +366,4 @@ class PlaceholderMapperSpec extends UnitSpec with MockFactory with BeforeAndAfte it should "return a String representation of the file access category NONE" in { testInstance.formatFileAccessRights(FileAccessRight.NONE) shouldBe "None" } - - "filesTable" should "give a mapping of files and checksums in the dataset" in { - val input = Seq( - FileItem("ABC", FileAccessRight.ANONYMOUS, Some("123")), - FileItem("DEF", FileAccessRight.KNOWN, Some("none")), - FileItem("GHI", FileAccessRight.RESTRICTED_GROUP, Some("")) - ) - - testInstance.filesTable(input).asScala should contain theSameElementsAs List( - Map(FilePath.keyword -> "ABC", FileChecksum.keyword -> "123", FileAccessibleTo.keyword -> "Anonymous").asJava, - Map(FilePath.keyword -> "DEF", FileChecksum.keyword -> checkSumNotCalculated, FileAccessibleTo.keyword -> "Known").asJava, - Map(FilePath.keyword -> "GHI", FileChecksum.keyword -> checkSumNotCalculated, FileAccessibleTo.keyword -> "Restricted group").asJava - ) - } } diff --git a/src/test/scala/nl/knaw/dans/easy/agreement/internal/VelocityTemplateResolverSpec.scala b/src/test/scala/nl/knaw/dans/easy/agreement/internal/VelocityTemplateResolverSpec.scala index 345fce3..b9b7432 100644 --- a/src/test/scala/nl/knaw/dans/easy/agreement/internal/VelocityTemplateResolverSpec.scala +++ b/src/test/scala/nl/knaw/dans/easy/agreement/internal/VelocityTemplateResolverSpec.scala @@ -15,19 +15,19 @@ */ package nl.knaw.dans.easy.agreement.internal -import java.io.{ByteArrayOutputStream, File} +import java.io.{ ByteArrayOutputStream, File } import java.util.Properties import nl.knaw.dans.easy.agreement.UnitSpec import org.apache.velocity.exception.MethodInvocationException import org.scalatest.exceptions.TestFailedException -import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll} +import org.scalatest.{ BeforeAndAfter, BeforeAndAfterAll } -import scala.util.{Failure, Success} +import scala.util.{ Failure, Success } class VelocityTemplateResolverSpec extends UnitSpec with BeforeAndAfter with BeforeAndAfterAll { - implicit val parameters = new BaseParameters(new File(testDir, "template"), null, false, 3) + implicit val parameters = new BaseParameters(new File(testDir, "template"), null, false) before { new File(getClass.getResource("/velocity/").toURI).copyDir(parameters.templateResourceDir) @@ -39,7 +39,7 @@ class VelocityTemplateResolverSpec extends UnitSpec with BeforeAndAfter with Bef override def afterAll: Unit = testDir.getParentFile.deleteDirectory() - private val keyword: KeywordMapping = new KeywordMapping { val keyword: String = "name" } + private val keyword: KeywordMapping = new KeywordMapping {val keyword: String = "name" } private val testProperties = { val p = new Properties() @@ -69,7 +69,7 @@ class VelocityTemplateResolverSpec extends UnitSpec with BeforeAndAfter with Bef templateCreator.createTemplate(baos, map) shouldBe a[Success[_]] - new String(baos.toByteArray) should include ("

    hello world

    ") + new String(baos.toByteArray) should include("

    hello world

    ") } it should "fail if not all placeholders are filled in" in { @@ -80,7 +80,7 @@ class VelocityTemplateResolverSpec extends UnitSpec with BeforeAndAfter with Bef val res = templateCreator.createTemplate(baos, map) res shouldBe a[Failure[_]] - (the [MethodInvocationException] thrownBy res.get).getMessage should include ("$name") + (the[MethodInvocationException] thrownBy res.get).getMessage should include("$name") new String(baos.toByteArray) should not include "

    hello world

    " }